Crawl Pages

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

The ID of the crawl to retrieve the list of URLs for.

Optional integer representing the index to start fetching content from. Useful to paginate until all URLs are fetched. Start with 0, then provide response['cursor'] value of the last request.

Optional integer to limit the number of results returned. Recommended 10-50 results at a time. Paginated using cursor. Maximum 10MB of content can be fetched in a single request.

An optional search query to sort the results by relevance. Uses the original search_query by default if provided.

Deprecated: Use /retrieve endpoint with retrieve_id.

Array of formats to fetch (e.g., ["html", "markdown"]).

Crawl ID

To be passed in the query in next request to get the next items.

The kind of object. "crawl" for this endpoint.

in_progress or completed