/v1/crawls endpoint you can crawl a website and get the content from all the pages.
- Crawl a website and get the content from all subpages (or limit the depth of the crawl)
- Use special patterns to crawl specific pages (e.g.
/blog/**) - Pass a
webhookURL to get notified when the crawl is completed - Search query to only find specific pages and sort by relevance
Quick reference for AI agents
Quick reference for AI agents
- Endpoint:
POST https://api.olostep.com/v1/crawls(auth:Authorization: Bearer <YOUR_API_KEY>) - Required:
start_url,max_pages - Common options:
include_urls/exclude_urls(glob patterns such as/blog/**),max_depth,search_query+top_n,webhook,scrape_options(formats,parser),include_subdomain - Returns:
id,status(in_progressorcompleted),pages_count - Sync or async: Async — poll
GET /v1/crawls/{crawl_id}untilstatusiscompleted(or setwebhook), list pages withGET /v1/crawls/{crawl_id}/pages(followcursor), then fetch each page withGET /v1/retrieve?retrieve_id=... - How it ends: a crawl ends as
completedeven if it finds 0 pages, so checkpages_count. Also stop polling ifstatusisfailed. If a retrieved page hassize_exceeded: true(over 6MB), its content fields are empty; download it from the*_hosted_urlfields instead. - SDK / CLI:
client.crawls.create(start_url=..., max_pages=...)thencrawl.pages()(Python) ·client.crawls.create({ url, maxPages })(Node) ·olostep crawl <url> --max-pages 100 - Use instead: URLs only → Map · a known list of URLs → Batches
Installation
Start a crawl
Provide the starting URL, include/exclude URL globs, andmax_pages. Optional: max_depth, include_external, include_subdomain, search_query, top_n, webhook, timeout.
crawl object in response. The crawl object has a few properties like id and status, which you can use to track the crawl.
Check crawl status
Poll the crawl to track progress untilstatus is completed.
webhook URL when starting the crawl to be notified when the crawl is completed.
List pages (paginate/stream with cursor)
Fetch pages and iterate usingcursor and limit. Works while the crawl is in_progress or completed.
Search query (limit to top N relevant)
Usesearch_query at start, and optionally filter listing with search_query. Limit per-page exploration with top_n.
Retrieve content
Use each page’sretrieve_id with /v1/retrieve to fetch html_content and/or markdown_content.
Notes
- Pagination is cursor-based; repeat requests until
cursoris absent. - Content fields on
/v1/crawls/{crawl_id}/pagesare deprecated; prefer/v1/retrieve. - Webhooks: set
webhookto receive a POST when the crawl completes.
Storage
Stored files for a crawl are kept for 7 days by default. Pass astorage object on POST /v1/crawls to keep them longer, or indefinitely. The setting is passed to every page scrape in the crawl, so it covers the content you fetch later with retrieve_id.
Omitting
storage, or sending storage without expires_in, uses 7d. Both the create response and GET /v1/crawls/{crawl_id} include the value in effect:
cURL
400: