Skip to main content
Through the Olostep /v1/batches endpoint you can process up to 10k URLs in one single batch. A batch takes around 5-8 mins. Use it to extract content or structured data at scale in an async way.
  • Submit up to 10k URLs per batch
  • A single batch takes around 5–8 minutes, regardless of batch size. Batch is an async endpoint
  • Run many batches in parallel to scale to millions of concurrent requests.
  • Use parsers to return structured JSON, or retrieve markdown/html via /v1/retrieve
  • If you want to get results with low latency or in a sync way, use the scrape endpoint and send many concurrent requests instead.
For API details see the Batch Endpoint API Reference.
Note: For new accounts, batches are limited to 100 items per batch. To lift this limitation, please contact us at info@olostep.com or reach out on Slack.

Installation

Start a batch

Provide an array of items with a custom_id and url. These are the URLs that will be processed in the batch, the custom_id is an internal unique identifier for the URL. Optionally pass parser or country. Through the parser parameter you can specify the parser to use for the batch, this will return structured JSON from the pages.
Since Olostep follows an object-oriented approach, you will receive a batch object in response. The batch object has a few properties like id and status.

Check batch status

Poll until status is completed. You can also check the completed_urls property to see how many URLs have been processed.

Retrieve content

Use the retrieve_id from each item with /v1/retrieve to fetch html_content, markdown_content, or json_content.

List items (paginate with cursor)

Fetch items using cursor and limit. Prefer using /v1/retrieve with retrieve_id for content.

Response Format

When you run the provided example code, you will receive a response like the following
Since we passed the parser to retrieve the structured JSON and we are printing only that, the response contains:
  • json_content with structured search results including:
  • searchParameters: Information about the search query
  • knowledgeGraph: Detailed information about the search subject (when available)
  • organic: List of search results with title, link, position, and snippet
  • peopleAlsoAsk: Related questions that users commonly search for
  • relatedSearches: Suggested related search queries
If you don’t want the structured JSON but simply the markdown or html you can retrieve those from the retrieve endpoint.

Webhooks

Instead of polling batch status, you can pass a webhook URL when you create the batch. Olostep sends an HTTP POST to that URL when the batch finishes (all items completed or failed). Your webhook endpoint must be publicly reachable over http:// or https://. It cannot point to localhost or private IP addresses. For the full payload shape, retry behavior, and best practices (respond quickly with 2xx, deduplicate using the event id), see Webhooks.
Parameter name: The canonical field is webhook. For backward compatibility, webhook_url is also accepted as an alias.
For batches, the batch.completed event includes the batch id, status, and item counts. Failed deliveries are retried automatically (up to 5 attempts over about 30 minutes with exponential backoff). Your handler must return a 2xx status within 30 seconds per attempt.

Metadata

Attach custom string key-value metadata to batches for tracking, filtering, and correlating jobs with your own systems (order IDs, project names, pipeline stage, and so on). Metadata follows the same rules as in our Metadata reference. You can set metadata at two levels when creating a batch:
  • Batch-levelmetadata on the request body (applies to the whole batch)
  • Item-levelmetadata on each object in the items array (per URL)
Metadata is returned on subsequent GET responses for that batch. You can merge-update batch metadata later with Update Batch (PATCH); see the metadata guide for add, overwrite, and delete behavior.
Type coercion: Numbers and booleans are converted to strings (for example 42"42", true"true"). Nested objects and arrays are rejected.
For complete examples and PATCH semantics, see Metadata.

Important Notes

If you want structured JSON you need to pass the specific parser to the API before making the requests. For example if you want to get the JSON from Google Search you will pass this parser "parser": {"id": "@olostep/google-search"} You can create your own parsers to retrieve the data you want from any page. Reach out to info@olostep.com to learn more.

Conclusion

The batch endpoint is useful if you need to get data from many URLs in a short period of time. You need to already have the list of urls you want to get the data from. Common applications can be:
  • Price tracking services that monitor multiple e-commerce sites for price changes on products
  • Website monitoring tools that check for content updates across numerous pages
  • Data aggregation for concert organizers tracking ticket availability across multiple venues
  • Search engines gathering and indexing content from many websites simultaneously
  • News aggregators collecting articles from various publications
  • Real estate platforms monitoring property listings across multiple sites
  • Job boards aggregating openings from company career pages
  • Financial data services tracking stock prices and market information
  • Social media monitoring tools analyzing mentions across different platforms
  • Academic research gathering data from multiple sources for analysis
It’s better to use the batch endpoint if you want the content from 100 to 10k urls at a time. If you need to scrape less than 50 urls at a time we recommended using the scrape endpoint and submit the urls in parallel since it’s faster than the batch endpoint.

Pricing

Batch costs 1 credit per URL.