# Create Answer
Source: https://docs.olostep.com/api-reference/answers/create
openapi/answers.json POST /v1/answers
The AI will perform actions like searching and browsing web pages to find the answer to the provided task.
Execution time is 3-30s depending upon complexity.
For longer tasks, use the agent endpoint instead. See [Agent feature](/features/agents). See [Answers feature](/features/answers).
# Get Answer
Source: https://docs.olostep.com/api-reference/answers/get
openapi/answers.json GET /v1/answers/{answer_id}
This endpoint retrieves a previously completed answer by its ID.
# Create Batch
Source: https://docs.olostep.com/api-reference/batches/create
openapi/batches.json POST /v1/batches
Starts a new batch. You receive an `id` that you can use to track the progress of the batch as shown [here](/api-reference/batches/info). Note: Processing time is constant regardless of batch size
**Get notified on completion:** Pass the `webhook` parameter with your endpoint URL to receive an HTTP POST when the batch completes. See [Webhooks](/api-reference/common/webhooks) for details.
**Attach custom data:** Use the `metadata` parameter to store key-value pairs. Supported at two levels:
* **Batch-level** — on the request body
* **Item-level** — on each item in the `items` array
See [Metadata](/api-reference/common/metadata) for details.
# Batch Info
Source: https://docs.olostep.com/api-reference/batches/info
openapi/batches.json GET /v1/batches/{batch_id}
Retrieves the status and progress information about a batch. To retrieve the content for a batch, see [here](/api-reference/batches/items)
**Attach custom data:** Use the `metadata` parameter to store key-value pairs with your batch. See [Metadata](/api-reference/common/metadata) for details.
# Batch Items
Source: https://docs.olostep.com/api-reference/batches/items
openapi/batches.json GET /v1/batches/{batch_id}/items
Retrieves the list of items processed for a batch. You can then use the `retrieve_id` to get the content with the Retrieve Endpoint
**Metadata:** If you attached `metadata` to individual items when [creating the batch](/api-reference/batches/create), it will be returned with each item in the response.
# Update Batch
Source: https://docs.olostep.com/api-reference/batches/update
openapi/batches.json PATCH /v1/batches/{batch_id}
Updates the metadata for a specific batch. Only metadata can be updated after batch creation.
**Merge semantics:** Metadata updates follow Stripe's approach — new keys are added, existing keys are updated, and keys set to empty string `""` are deleted.
# Get credit info
Source: https://docs.olostep.com/api-reference/billing/credits-info
openapi/billing.json GET /user/credits/info
Returns the authenticated team credit balance, per-lot breakdown, active subscription, and whether usage is allowed.
For a guided overview, see [Balance & billing](/balance-billing/balance-and-billing).
# Purchase top-up
Source: https://docs.olostep.com/api-reference/billing/purchase-topup
openapi/billing.json POST /user/purchase-topup
Charge a saved card and purchase credits in one step. No Checkout redirect or embedded payment UI.
**`credits` must be one of:** `10000`, `20000`, `80000`, or `100000` — that is **10K**, **20K**, **80K**, or **100K** credits.
For a guided overview, see [Balance & billing](/balance-billing/balance-and-billing).
# Metadata
Source: https://docs.olostep.com/api-reference/common/metadata
Attach custom key-value pairs to API resources
**Currently available for [Batches](/api-reference/batches/create).** Support for scrapes, crawls, maps, and answers is coming soon.
Metadata allows you to attach custom key-value pairs to Olostep resources. This is useful for tracking, filtering, organizing, and storing context alongside your API requests.
Metadata follows [Stripe's approach](https://stripe.com/docs/api/metadata) — simple, flexible, and consistent across all endpoints.
***
## Use Cases
Link resources to internal systems with order IDs, customer IDs, or project names.
Tag resources for easy retrieval and filtering in your application.
Store pipeline stage, priority level, or processing instructions.
Record who initiated a request, timestamps, or version information.
***
## Adding Metadata on Create
Include the `metadata` parameter when creating a resource:
```json Request Body theme={null}
{
"url": "https://example.com",
"metadata": {
"order_id": "12345",
"customer_name": "John Doe",
"priority": "high",
"internal_ref": "proj-2024-001"
}
}
```
```python Python theme={null}
import requests
response = requests.post(
"https://api.olostep.com/v1/batches",
headers={"Authorization": "Bearer "},
json={
"items": [{"custom_id": "1", "url": "https://example.com"}],
"metadata": {
"project": "q4-analysis",
"team": "data-ops",
"priority": "high"
}
}
)
```
```javascript Node.js theme={null}
const response = await fetch("https://api.olostep.com/v1/batches", {
method: "POST",
headers: {
"Authorization": "Bearer ",
"Content-Type": "application/json"
},
body: JSON.stringify({
items: [{ custom_id: "1", url: "https://example.com" }],
metadata: {
project: "q4-analysis",
team: "data-ops",
priority: "high"
}
})
});
```
Metadata is returned in all subsequent GET responses for that resource.
***
## Validation Rules
| Constraint | Limit | Error Example |
| ------------ | ------------------ | ------------------------------------------------------------------------- |
| Maximum keys | 50 | `"Metadata can have a maximum of 50 keys. You provided 51 keys."` |
| Key length | 40 characters | `"Metadata key \"my_very_long_key_name...\" exceeds 40 character limit."` |
| Key format | No square brackets | `"Metadata key \"items[0]\" cannot contain square brackets ([ or ])."` |
| Value length | 500 characters | `"Metadata value for key \"description\" exceeds 500 character limit."` |
| Value type | Strings only | `"Metadata value for key \"count\" must be a string. Got object."` |
**Type Coercion**: Numbers and booleans are automatically converted to strings.
* `42` → `"42"`
* `true` → `"true"`
* `3.14` → `"3.14"`
Objects and arrays are rejected.
***
## Updating Metadata (PATCH)
**Currently available for:** [Batches](/api-reference/batches/update) only.
Crawls, Scrapes, Maps, and Answers do not yet support updating metadata after creation.
You can update metadata on existing batches using the [PATCH endpoint](/api-reference/batches/update). Updates use merge behavior.
### Update Operations
New keys are added while preserving existing ones.
```bash theme={null}
curl -X PATCH "https://api.olostep.com/v1/batches/batch_abc123" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"metadata": {"new_key": "new_value"}}'
```
**Before:** `{"project": "alpha"}`\
**After:** `{"project": "alpha", "new_key": "new_value"}`
Existing keys are overwritten with new values.
```bash theme={null}
curl -X PATCH "https://api.olostep.com/v1/batches/batch_abc123" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"metadata": {"project": "beta"}}'
```
**Before:** `{"project": "alpha", "priority": "high"}`\
**After:** `{"project": "beta", "priority": "high"}`
Set a key to `null` or `""` (empty string) to delete it.
```bash theme={null}
curl -X PATCH "https://api.olostep.com/v1/batches/batch_abc123" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"metadata": {"priority": null}}'
```
**Before:** `{"project": "alpha", "priority": "high"}`\
**After:** `{"project": "alpha"}`
Set the entire metadata field to `null` or `""` to remove all keys.
```bash theme={null}
curl -X PATCH "https://api.olostep.com/v1/batches/batch_abc123" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"metadata": null}'
```
**Before:** `{"project": "alpha", "priority": "high"}`\
**After:** `{}`
Add, update, and delete keys in a single request.
```bash theme={null}
curl -X PATCH "https://api.olostep.com/v1/batches/batch_abc123" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"metadata": {"project": "gamma", "new_field": "value", "old_field": null}}'
```
**Before:** `{"project": "alpha", "old_field": "remove_me"}`\
**After:** `{"project": "gamma", "new_field": "value"}`
### PATCH Behavior Summary
| Operation | Request | Result |
| ---------- | ----------------------------------------- | ----------------------------- |
| Add key | `{"metadata": {"new": "value"}}` | Key added, others preserved |
| Update key | `{"metadata": {"existing": "new_value"}}` | Key updated, others preserved |
| Delete key | `{"metadata": {"key": null}}` | Key removed, others preserved |
| Delete key | `{"metadata": {"key": ""}}` | Key removed, others preserved |
| Clear all | `{"metadata": null}` | All keys removed |
| Clear all | `{"metadata": ""}` | All keys removed |
| No-op | `{"metadata": {}}` | No changes |
# Object-Oriented API
Source: https://docs.olostep.com/api-reference/common/object-oriented
Understanding how Olostep API objects work together
Olostep's API is designed around objects. Understanding this design helps you build more effective integrations.
***
## Everything is an Object
Every resource in Olostep is an object with a unique identifier. Whether you create it via the API, SDK, or dashboard — you get back an object you can reference, update, and query.
| Resource | Object ID Format | Example |
| -------- | ---------------- | ----------------- |
| Scrape | `scrape_*` | `scrape_abc123` |
| Batch | `batch_*` | `batch_xyz789` |
| Crawl | `crawl_*` | `crawl_def456` |
| Map | `map_*` | `map_ghi012` |
| Answer | `answer_*` | `answer_jkl345` |
| Search | `search_*` | `search_mno678` |
| Monitor | `monitor_*` | `monitor_mno678` |
| File | `file_*` | `file_mno678` |
| Schedule | `schedule_*` | `schedule_pqr901` |
***
## Objects Can Have Lifecycles
Some Olostep objects track state through a `status` field. This state machine pattern lets you know exactly where each resource is in its lifecycle.
### Batches
Batches have two levels of status: the **batch** itself and individual **items**.
**Batch Status:**
```
in_progress → completed
```
| Status | Description |
| ------------- | ---------------------- |
| `in_progress` | URLs are being scraped |
| `completed` | Processing finished |
**Batch-level failures are extremely rare.** Batches almost always complete — even if some URLs fail, the batch itself reaches `completed` status. In the rare case of a catastrophic infrastructure failure (e.g., LLM service outage during enrichment), the batch may fail. This affects less than 0.01% of batches.
**Item Status:**
Each URL in a batch is tracked as an individual item with its own status:
| Status | Description |
| --------- | ------------------------ |
| `success` | URL scraped successfully |
| `failed` | URL could not be scraped |
Items can fail due to:
* URL is blocked or returns an error
* Parser output missing
* Network/fetch errors
Failed items include an `error` object with `code` and `message` explaining the failure. The batch still completes — check each item's status when processing results.
### Crawls
```
in_progress → completed
```
| Status | Description |
| ------------- | ---------------------------------------- |
| `in_progress` | Actively discovering and processing URLs |
| `completed` | Crawling finished |
**Crawls always complete.** Even if a crawl finds 0 URLs (due to robots.txt blocking or invalid start URL), the crawl status will be `completed`. Check the `pages_count` field to verify results.
### Monitors
Monitors are long-lived objects with a richer lifecycle than one-shot resources:
```
provisioning → active ⇄ paused
↘ failed
```
| Status | Description |
| -------------- | --------------------------------------------------------------- |
| `provisioning` | Agent, spec, planner, and schedule are being set up |
| `active` | Schedule enabled; runs execute on `schedule.frequency` |
| `paused` | Schedule disabled via `POST /v1/monitors/:monitor_id/pause` |
| `failed` | Provisioning or schedule update failed (`error_message` is set) |
| `deleted` | Soft-deleted via `DELETE /v1/monitors/:monitor_id` |
Creating a monitor returns HTTP `202` with `status: provisioning`. The monitor becomes `active` once planning resolves its tracked targets — poll `GET /v1/monitors/:monitor_id` or stream provisioning events with `?stream=1`. Only `active` monitors can be paused, and only `paused` monitors can be resumed. Updates return `409` while the monitor is still `provisioning`.
***
## Retrieve Pattern
Many objects produce content that can be retrieved later. The `retrieve_id` pattern lets you fetch content without re-processing.
```bash theme={null}
# Get content using retrieve_id
curl "https://api.olostep.com/v1/retrieve?retrieve_id=6h89o8u1kt" \
-H "Authorization: Bearer "
```
This pattern is used by:
* **Batch items** — Each processed URL gets a `retrieve_id`
* **Crawl pages** — Each crawled page gets a `retrieve_id`
The `/v1/retrieve` endpoint accepts `formats` parameter to specify which content types to return (`html`, `markdown`, `json`, `text`).
***
## Webhooks: Event-Driven Updates
Instead of polling for status changes, configure [webhooks](/api-reference/common/webhooks) to receive events when objects change state.
```json theme={null}
{
"event": "batch.completed",
"data": {
"id": "batch_xyz789",
"status": "completed",
"items_total": 100,
"items_completed": 100
}
}
```
***
## Metadata: Your Data Alongside Ours
Attach custom key-value pairs to objects using [metadata](/api-reference/common/metadata). This lets you link Olostep resources to your internal systems.
```json theme={null}
{
"items": [{"url": "https://example.com"}],
"metadata": {
"order_id": "12345",
"customer": "acme-corp"
}
}
```
***
## Summary
| Concept | Description |
| -------------- | ----------------------------------------------- |
| **Objects** | Every resource has a unique ID and is queryable |
| **Lifecycles** | Track progress via `status` field |
| **Retrieve** | Fetch content later with `retrieve_id` |
| **Webhooks** | Get notified when state changes |
| **Metadata** | Attach your own data to any object |
# Webhooks
Source: https://docs.olostep.com/api-reference/common/webhooks
Receive real-time notifications when async operations complete
**We're actively expanding webhook support.**
Just landed: automatic retries with exponential backoff — failed deliveries are now retried up to 5 times over 30 minutes.
**Coming soon:**
* Team-wide default webhook URLs
* Cryptographic signatures for payload verification
Want early access? Reach out at [info@olostep.com](mailto:info@olostep.com) or join our [Slack community](https://olostep-users.slack.com/join/shared_invite/zt-2bfddyi8h-JzfjOgavg~98DJ1om1B5Lg).
## Overview
Webhooks deliver real-time HTTP POST notifications to your server when long-running operations complete. Instead of polling for status, your application receives instant updates.
### Use Cases
Get notified when batches or crawls complete instead of polling
Automatically trigger downstream processing when data is ready
Send alerts to Slack, email, or other systems on completion
Keep your database in sync with Olostep results
## Supported Events
Fired when a batch finishes processing (all items completed or failed).
```json theme={null}
{
"id": "event_a1b2c3d4e5f6g7h8",
"object": "event.batch.completed",
"timestamp": 1737570000000,
"delivery_attempt": "1/5",
"data": {
"id": "batch_xyz123",
"object": "batch",
"status": "completed",
"items_total": 100,
"items_completed": 98,
"items_failed": 2,
"created_at": "2024-01-15T10:00:00Z",
"completed_at": "2024-01-15T10:05:32Z"
}
}
```
Fired when a crawl finishes and all discovered pages have been processed.
```json theme={null}
{
"id": "event_x9y8z7w6v5u4t3s2",
"object": "event.crawl.completed",
"timestamp": 1737570000000,
"delivery_attempt": "1/5",
"data": {
"id": "crawl_abc789",
"object": "crawl",
"status": "completed",
"start_url": "https://example.com",
"urls_count": 87,
"max_pages": 100,
"max_depth": 3,
"actual_max_depth": 3,
"start_epoch": 1737569500000,
"start_date": "2024-01-15"
}
}
```
***
## Setting Up Webhooks
Pass `webhook` when creating a resource. This URL receives the completion notification.
**Parameter name:** The canonical parameter is `webhook`. For backward compatibility, `webhook_url` is also accepted as an alias.
```python Python theme={null}
import requests
# Batch example
response = requests.post(
"https://api.olostep.com/v1/batches",
headers={"Authorization": "Bearer "},
json={
"items": [
{"url": "https://example.com/page1", "custom_id": "1"},
{"url": "https://example.com/page2", "custom_id": "2"}
],
"webhook": "https://your-server.com/webhooks/olostep"
}
)
# Crawl example
response = requests.post(
"https://api.olostep.com/v1/crawls",
headers={"Authorization": "Bearer "},
json={
"start_url": "https://example.com",
"max_pages": 50,
"webhook": "https://your-server.com/webhooks/olostep"
}
)
```
```js Node theme={null}
// Batch example
const batchResponse = await fetch('https://api.olostep.com/v1/batches', {
method: 'POST',
headers: {
'Authorization': 'Bearer ',
'Content-Type': 'application/json'
},
body: JSON.stringify({
items: [
{ url: 'https://example.com/page1', custom_id: "1" },
{ url: 'https://example.com/page2', custom_id: "2" }
],
webhook: 'https://your-server.com/webhooks/olostep'
})
});
// Crawl example
const crawlResponse = await fetch('https://api.olostep.com/v1/crawls', {
method: 'POST',
headers: {
'Authorization': 'Bearer ',
'Content-Type': 'application/json'
},
body: JSON.stringify({
start_url: 'https://example.com',
max_pages: 50,
webhook: 'https://your-server.com/webhooks/olostep'
})
});
```
```bash cURL theme={null}
# Batch example
curl -X POST "https://api.olostep.com/v1/batches" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"items": [
{"url": "https://example.com/page1", "custom_id": "1"},
{"url": "https://example.com/page2", "custom_id": "2"}
],
"webhook": "https://your-server.com/webhooks/olostep"
}'
# Crawl example
curl -X POST "https://api.olostep.com/v1/crawls" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"start_url": "https://example.com",
"max_pages": 50,
"webhook": "https://your-server.com/webhooks/olostep"
}'
```
***
## Webhook Payload
All webhook payloads follow a unified envelope structure:
```json theme={null}
{
"id": "event_a1b2c3d4e5f6g7h8",
"object": "event.batch.completed",
"timestamp": 1737570000000,
"delivery_attempt": "1/5",
"data": {
"id": "batch_xyz123",
"object": "batch",
"status": "completed",
"items_total": 100,
"items_completed": 98,
"items_failed": 2
}
}
```
### Envelope Fields
| Field | Description |
| ------------------ | ------------------------------------------------------ |
| `id` | Event ID — **same across all retry attempts** |
| `object` | Event type (e.g., `event.batch.completed`) |
| `timestamp` | When this delivery attempt was sent (epoch ms) |
| `delivery_attempt` | Current attempt / max attempts (e.g., `1/5`, `3/5`) |
| `data` | The actual resource data (same format as API response) |
Use the `id` field to deduplicate webhook deliveries in your receiver. The same event ID appears in all retry attempts.
***
## Retry Behavior
Failed webhook deliveries are automatically retried with exponential backoff over a 30-minute window:
| Attempt | Delay Before Attempt | Cumulative Time |
| ------- | -------------------- | --------------- |
| 1 | Immediate | 0 min |
| 2 | \~2 min | \~2 min |
| 3 | \~4 min | \~6 min |
| 4 | \~7 min | \~13 min |
| 5 | \~15 min | \~28 min |
**Total retry window:** 30 minutes\
**Per-request timeout:** 30 seconds
### What Counts as Success
Your endpoint must return a `2xx` status code within 30 seconds. Any other response triggers a retry.
| Response | Result |
| ------------------ | ----------------------------------- |
| `200 OK` | ✅ Delivered |
| `201 Created` | ✅ Delivered |
| `301 Redirect` | ❌ Retry (we don't follow redirects) |
| `400 Bad Request` | ❌ Retry |
| `500 Server Error` | ❌ Retry |
| Timeout (>30s) | ❌ Retry |
| Connection refused | ❌ Retry |
***
## Best Practices
Return `200 OK` immediately and process the webhook asynchronously. If your processing takes longer than 30 seconds, we'll retry — causing duplicate deliveries.
```python theme={null}
from queue import Queue
import threading
webhook_queue = Queue()
@app.route('/webhooks/olostep', methods=['POST'])
def handle_webhook():
# Queue for async processing
webhook_queue.put(request.json)
# Return immediately
return 'OK', 200
def process_webhooks():
while True:
event = webhook_queue.get()
# Slow processing happens here
process_event(event)
threading.Thread(target=process_webhooks, daemon=True).start()
```
Use the `id` field to deduplicate. Store processed event IDs and skip duplicates.
```python theme={null}
processed_events = set() # Use Redis/DB in production
def handle_event(event):
if event['id'] in processed_events:
return # Already processed
# Process the event
process_batch_completed(event['data'])
# Mark as processed
processed_events.add(event['id'])
```
Log all webhook receipts for debugging. Include the event ID, timestamp, and processing result.
```python theme={null}
import logging
@app.route('/webhooks/olostep', methods=['POST'])
def handle_webhook():
event = request.json
logging.info(f"Webhook received: id={event['id']} type={event['object']} attempt={event['delivery_attempt']}")
try:
process_event(event)
logging.info(f"Webhook processed: id={event['id']}")
except Exception as e:
logging.error(f"Webhook failed: id={event['id']} error={e}")
raise
return 'OK', 200
```
Always use HTTPS for webhook endpoints. HTTP endpoints are vulnerable to eavesdropping and man-in-the-middle attacks.
***
## Troubleshooting
1. Verify the `webhook` parameter was included in your request
2. Verify your endpoint is publicly accessible (not localhost)
3. Check your server logs for incoming requests
4. Ensure you're returning a `2xx` status code
This is expected during retries. Implement idempotent handling using the `id` field:
```python theme={null}
def handle_event(event):
if already_processed(event['id']):
return # Skip duplicate
process_event(event)
mark_processed(event['id'])
```
Your endpoint must respond within 30 seconds. Process webhooks asynchronously:
```python theme={null}
@app.route('/webhooks', methods=['POST'])
def webhook():
queue.enqueue(process_webhook, request.json)
return 'OK', 200 # Respond immediately
```
***
## Coming Soon
Configure a default webhook URL in your account settings. All requests will use this URL unless overridden.
Cryptographic signatures (HMAC-SHA256) to verify webhook payloads came from Olostep.
Want early access to these features? Contact us at [info@olostep.com](mailto:info@olostep.com) or join our [Slack community](https://olostep-users.slack.com/join/shared_invite/zt-2bfddyi8h-JzfjOgavg~98DJ1om1B5Lg).
# Create Crawl
Source: https://docs.olostep.com/api-reference/crawls/create
openapi/crawls.json POST /v1/crawls
Starts a new crawl. You receive a `id` to track the progress. The operation may take 1-10 mins depending upon the site and depth and pages parameters.
**Get notified on completion:** Pass the `webhook` parameter with your endpoint URL to receive an HTTP POST when the crawl completes. See [Webhooks](/api-reference/common/webhooks) for details.
# Crawl Info
Source: https://docs.olostep.com/api-reference/crawls/info
openapi/crawls.json GET /v1/crawls/{crawl_id}
Fetches information about a specific crawl.
# Crawl Pages
Source: https://docs.olostep.com/api-reference/crawls/pages
openapi/crawls.json GET /v1/crawls/{crawl_id}/pages
Fetches the list of pages for a specific crawl.
# Complete File Upload
Source: https://docs.olostep.com/api-reference/files/complete
openapi/files.json POST /v1/files/{file_id}/complete
Complete the file upload process and validate the uploaded JSON file. This endpoint verifies the file exists, validates its JSON format, and updates the file status to completed.
# Get File Content
Source: https://docs.olostep.com/api-reference/files/content
openapi/files.json GET /v1/files/{file_id}/content
Download the content of a completed file. Returns the JSON file content.
# Create File Upload
Source: https://docs.olostep.com/api-reference/files/create
openapi/files.json POST /v1/files
Generate a pre-signed URL for uploading a JSON file. After uploading, you must call the [complete endpoint](/api-reference/files/complete) to finalize the upload.
# Delete File
Source: https://docs.olostep.com/api-reference/files/delete
openapi/files.json DELETE /v1/files/{file_id}
Delete a file and its associated data from storage.
# Get File
Source: https://docs.olostep.com/api-reference/files/get
openapi/files.json GET /v1/files/{file_id}
Retrieve metadata for a file by its ID.
# List Files
Source: https://docs.olostep.com/api-reference/files/list
openapi/files.json GET /v1/files
List all completed files for your team. Optionally filter by purpose.
# Create Map
Source: https://docs.olostep.com/api-reference/maps/create
openapi/maps.json POST /v1/maps
This endpoint allows users to get all the urls on a certain website. It can take up to 120 seconds for complex websites. For large websites, results are paginated using cursor-based pagination
# Get Map
Source: https://docs.olostep.com/api-reference/maps/get
openapi/maps.json GET /v1/maps/{map_id}
Retrieve a previously completed map by its ID.
# Stream Monitor Agent Logs
Source: https://docs.olostep.com/api-reference/monitors/agent-logs
openapi/monitors.json GET /v1/monitors/{monitor_id}/agent-logs
Tails CloudWatch logs for the monitor agent, filtered to this monitor_id. Requires ?stream=1 or Accept: text/event-stream.
# Create Monitor
Source: https://docs.olostep.com/api-reference/monitors/create
openapi/monitors.json POST /v1/monitors
Creates a monitor from a natural-language query. Provisions a shadow agent, generates a workflow spec, queues DAG planning, and schedules recurring runs. Configure delivery with notification.channels and/or webhook. Returns 202 with status provisioning, or stream progress with ?stream=1.
# Delete Monitor
Source: https://docs.olostep.com/api-reference/monitors/delete
openapi/monitors.json DELETE /v1/monitors/{monitor_id}
Soft-deletes a monitor and removes its related scheduling/shadow-agent resources.
# List Monitor Events
Source: https://docs.olostep.com/api-reference/monitors/events
openapi/monitors.json GET /v1/monitors/{monitor_id}/events
Lists paginated snapshot events (newest first) with pre-signed snapshot_url values. Supports count_only=true for total_count only.
# Get Monitor
Source: https://docs.olostep.com/api-reference/monitors/get
openapi/monitors.json GET /v1/monitors/{monitor_id}
Retrieves a monitor by ID, including last_run and total_count. Use include-diagram=true for a Mermaid DAG diagram and include_total_count=false to omit the snapshot count.
# List Monitors
Source: https://docs.olostep.com/api-reference/monitors/list
openapi/monitors.json GET /v1/monitors
Retrieves all monitors for your team. Returns active monitors by default and can include deleted monitors when requested.
# Pause Monitor
Source: https://docs.olostep.com/api-reference/monitors/pause
openapi/monitors.json POST /v1/monitors/{monitor_id}/pause
Pauses a monitor by disabling future scheduled runs.
# Get Monitor Planning
Source: https://docs.olostep.com/api-reference/monitors/planning
openapi/monitors.json GET /v1/monitors/{monitor_id}/planning
Returns the FDA workflow spec and planner DAG for a monitor shadow agent.
# Resume Monitor
Source: https://docs.olostep.com/api-reference/monitors/resume
openapi/monitors.json POST /v1/monitors/{monitor_id}/resume
Resumes a paused monitor by re-enabling scheduled runs.
# Get Monitor Run
Source: https://docs.olostep.com/api-reference/monitors/run
openapi/monitors.json GET /v1/monitors/{monitor_id}/runs/{run_id}
Returns snapshot metadata and parsed agent log events for a single monitor run.
# Update Monitor
Source: https://docs.olostep.com/api-reference/monitors/update
openapi/monitors.json POST /v1/monitors/{monitor_id}
Updates frequency, metadata, notification, and/or webhook. Frequency changes recreate the schedule. Returns 409 while status is provisioning.
**Merge semantics:** Metadata updates follow Stripe's approach — new keys are added, existing keys are updated, and keys set to empty string `""` are deleted.
# Retrieve Content
Source: https://docs.olostep.com/api-reference/retrieve
openapi/utility.json GET /v1/retrieve
Retrieve content of processed batches and crawls urls.
# Create Schedule
Source: https://docs.olostep.com/api-reference/schedules/create
openapi/schedules.json POST /v1/schedules
Creates a new schedule to execute API calls at specified times. Supports both one-time executions and recurring schedules using cron expressions. You can also use natural language text to generate cron expressions automatically.
# Delete Schedule
Source: https://docs.olostep.com/api-reference/schedules/delete
openapi/schedules.json DELETE /v1/schedules/{schedule_id}
Deletes a schedule by its ID. This will stop any future executions and remove the schedule from EventBridge.
# Get Schedule
Source: https://docs.olostep.com/api-reference/schedules/get
openapi/schedules.json GET /v1/schedules/{schedule_id}
Retrieves a single schedule by its ID.
# List Schedules
Source: https://docs.olostep.com/api-reference/schedules/list
openapi/schedules.json GET /v1/schedules
Retrieves all schedules for your team. Returns a list of schedules with their configuration, status, and metadata.
# Create Scrape
Source: https://docs.olostep.com/api-reference/scrapes/create
openapi/scrapes.json POST /v1/scrapes
[Scrape](https://docs.olostep.com/features/scrapes) a url with provided configuration and get content.
**Optional caching:** Pass `max_age` (in seconds) to reuse a recent scrape with the same parameters instead of fetching the page again. Defaults to `0` (always fresh). In the dashboard playground, the default is 24 hours. See [Caching](/features/scrapes#caching) for details.
# Get Scrape
Source: https://docs.olostep.com/api-reference/scrapes/get
openapi/scrapes.json GET /v1/scrapes/{scrape_id}
Can be used to retrieve response for a scrape.
# Create Search
Source: https://docs.olostep.com/api-reference/searches/create
openapi/search.json POST /v1/searches
Search the web with a natural language query and get back a deduplicated list of relevant links with titles and descriptions.
It will search for the query semantically across the web and return results.
Optionally, pass `scrape_options` to also scrape every returned URL and embed `markdown_content` / `html_content` directly into each link. Page scrapes are billed automatically against your team via the underlying /v1/scrapes endpoint.
See [Search feature](/features/search).
# Get Search
Source: https://docs.olostep.com/api-reference/searches/get
openapi/search.json GET /v1/searches/{search_id}
Retrieve a previously completed search by its ID. Returns whatever was persisted at search time, including any scraped per-link content. Pure idempotent read — no re-scraping, no re-billing.
# Balance & billing
Source: https://docs.olostep.com/balance-billing/balance-and-billing
Check your credit balance and purchase top-ups programmatically.
Use the user billing endpoints to read your team's credit balance and buy top-ups with a saved payment method. Both endpoints require authentication with your [API key](/get-started/authentication). Invalid keys return **402**.
## Check credit balance
`GET /user/credits/info` returns the authenticated team's credit balance, a per-lot breakdown, active subscription details, and whether usage is allowed.
Use this endpoint to power billing widgets, usage dashboards, or pre-flight checks before running large jobs.
For API details see [Get credit info](/api-reference/billing/credits-info).
```python Python theme={null}
import requests
response = requests.get(
"https://api.olostep.com/user/credits/info",
headers={"Authorization": "Bearer "},
)
print(response.json())
```
```js Node theme={null}
const res = await fetch("https://api.olostep.com/user/credits/info", {
headers: { Authorization: "Bearer " },
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s "https://api.olostep.com/user/credits/info" \
-H "Authorization: Bearer "
```
### Response
```json theme={null}
{
"credits": 12500,
"breakdown": [
{
"purchase_kind": "Subscription",
"allocated_units": 10000,
"remaining_units": 8500,
"expiry_date": 1735689600
},
{
"purchase_kind": "Top-up",
"allocated_units": 5000,
"remaining_units": 4000,
"expiry_date": 1743465600
}
],
"active_subscription": {
"id": "SUB_PRO",
"display_name": "Pro",
"credits": 10000,
"created_at": 1704067200
},
"allow_usage": true
}
```
| Field | Description |
| --------------------- | ----------------------------------------------------------- |
| `credits` | Total remaining credits across all non-expired lots |
| `breakdown` | Per-lot detail: type, allocated and remaining units, expiry |
| `active_subscription` | Current plan (falls back to `SUB_BASE` if none is active) |
| `allow_usage` | Whether the team can still consume credits |
Each lot in `breakdown` has a `purchase_kind` of `Subscription`, `Top-up`, `Manual`, `Setup`, or `Pending`.
## Purchase a top-up
`POST /user/purchase-topup` charges a saved card on Stripe and buys credits in one step. There is no Checkout redirect or embedded payment UI.
Pass the credit amount in the request body:
```json theme={null}
{ "credits": 10000 }
```
| Field | Description |
| --------- | ----------------------------- |
| `credits` | Number of credits to purchase |
Supported values: **10,000**, **20,000**, **80,000**, and **100,000**.
For API details see [Purchase top-up](/api-reference/billing/purchase-topup).
```python Python theme={null}
import requests
response = requests.post(
"https://api.olostep.com/user/purchase-topup",
headers={
"Authorization": "Bearer ",
"Content-Type": "application/json",
},
json={"credits": 10000},
)
print(response.status_code)
print(response.json())
```
```js Node theme={null}
const res = await fetch("https://api.olostep.com/user/purchase-topup", {
method: "POST",
headers: {
Authorization: "Bearer ",
"Content-Type": "application/json",
},
body: JSON.stringify({ credits: 10000 }),
})
console.log(res.status)
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/user/purchase-topup" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"credits": 10000}'
```
### Requirements
* The team must have a Stripe customer with at least one saved card.
* Only one purchase attempt is allowed every **60 seconds** per team. If you hit the cooldown, the response is **429** with a `Retry-After` header.
### Responses
**200** — payment succeeded:
```json theme={null}
{
"success": true,
"payment_intent_id": "pi_xxx",
"credits": 10000
}
```
**202** — payment is still processing. Credits are added once Stripe confirms the payment:
```json theme={null}
{
"success": true,
"status": "processing",
"payment_intent_id": "pi_xxx",
"message": "Payment is processing. Credits will be added once the payment is confirmed.",
"credits": 10000
}
```
Credits are issued by the Stripe webhook after payment confirmation, not directly from the HTTP response. Treat **200** as payment success; poll `GET /user/credits/info` if you need to confirm the updated balance.
### Common errors
| Status | Error | When |
| ------ | ------------------------- | ------------------------------------------------ |
| 400 | `missing_topup_selector` | `credits` was not sent |
| 400 | `invalid_credits` | Credit amount is not in the allowed list |
| 400 | `no_stripe_customer` | Team has no Stripe customer |
| 400 | `no_payment_method` | No saved card on file |
| 402 | `payment_failed` | No saved card completed the charge |
| 429 | `purchase_topup_cooldown` | Another purchase was attempted within 60 seconds |
| 503 | `payment_status_unknown` | Ambiguous Stripe error; wait before retrying |
# Get the content of multiple websites in one go
Source: https://docs.olostep.com/examples/batch
Start a batch scrape to extract content from up to 100k URLs in 5-7 mins
## Overview
Olostep's [Batches](https://docs.olostep.com/api-reference/batches/create) endpoint allows you to start a batch of up to 10,000 URLs and get back the content in 5–7 minutes. You can start up to 10 batches at a time to extract content from 100,000 URLs in one go. If you need more scale, please reach out to us
This is useful if you already have the URLs you want to process —for example, to aggregate data for analysis, build a specialized search tool, or monitor multiple websites for changes.
In this guide, we’ll walk through how to start a batch with a list of URLs and retrieve the content in markdown format.
## Gist with Full Code
Here's all the code in one gist that you can copy and paste to try out batch scraping with Olostep:
[https://gist.github.com/olostep/e903f2e4fc28f8093b834b4df68b8031](https://gist.github.com/olostep/e903f2e4fc28f8093b834b4df68b8031)
In this gist we have shown how to start a batch with 5 google search queries, check the status, and retrieve the content for each item.
## Prerequisites
Before getting started, ensure you have the following:
* A valid Olostep API key. You can get one by signing up at [Olostep](https://www.olostep.com/dashboard).
* Python installed on your system.
* The `requests` and `hashlib` libraries (install `requests` with `pip install requests` if needed).
## Step 1: Create a Batch from Local URLs
If you already have a list of URLs you want to process, you can define them directly in your script. Otherwise, you can read them from a file or database.
```python filename="start_batch.py" theme={null}
import requests
import hashlib
API_KEY = "YOUR_API_KEY"
def create_hash_id(url):
return hashlib.sha256(url.encode()).hexdigest()[:16]
def compose_items_array():
urls = [
"https://www.google.com/search?q=nikola+tesla&gl=us&hl=en",
"https://www.google.com/search?q=alexander+the+great&gl=us&hl=en",
"https://www.google.com/search?q=google+solar+eclipse&gl=us&hl=en",
"https://www.google.com/search?q=crispr&gl=us&hl=en",
"https://www.google.com/search?q=genghis%20khan&gl=us&hl=en"
]
return [{"custom_id": create_hash_id(url), "url": url} for url in urls]
def start_batch(items):
payload = {
"items": items
}
headers = {"Authorization": f"Bearer {API_KEY}"}
response = requests.post(
"https://api.olostep.com/v1/batches",
headers=headers,
json=payload
)
return response.json()["id"]
if __name__ == "__main__":
items = compose_items_array()
batch_id = start_batch(items)
print("Batch started. ID:", batch_id)
```
## Step 2: Monitor Batch Status
Once the batch is started, you can monitor its status using the `batch_id` that is returned when you start the batch
```python filename="check_status.py" theme={null}
import requests
def check_batch_status(batch_id):
headers = {"Authorization": f"Bearer {API_KEY}"}
response = requests.get(
f"https://api.olostep.com/v1/batches/{batch_id}",
headers=headers
)
return response.json()["status"]
```
You can poll the status every few seconds (e.g. 10 seconds) until the batch is complete:
```python theme={null}
import time
def recursive_check(batch_id):
status = check_batch_status(batch_id)
print("Status:", status)
if status == "completed":
print("Batch is complete!")
else:
time.sleep(60)
recursive_check(batch_id)
```
## Step 3: Retrieve Completed Items
Once the batch is marked complete, fetch the processed items.
```python theme={null}
import requests
def get_completed_items(batch_id):
headers = {"Authorization": f"Bearer {API_KEY}"}
response = requests.get(
f"https://api.olostep.com/v1/batches/{batch_id}/items",
headers=headers
)
return response.json()["items"]
```
Each item will include a `retrieve_id` which you can use to fetch the scraped content.
```python theme={null}
items = get_completed_items(batch_id)
for item in items:
print(f"URL: {item['url']}\nCustom ID: {item['custom_id']}\nRetrieve ID: {item['retrieve_id']}\n---")
```
## Step 4: Retrieve the Content
Use the `retrieve_id` to get the extracted content in markdown, html or json. Here is an example to retrieve the content in markdown format:
```python filename="retrieve_content.py" theme={null}
def retrieve_content(retrieve_id):
url = "https://api.olostep.com/v1/retrieve"
headers = {"Authorization": f"Bearer {API_KEY}"}
params = {"retrieve_id": retrieve_id}
response = requests.get(url, headers=headers, params=params)
return response.json()
# Example usage:
items = get_completed_items(batch_id)
for item in items:
content = retrieve_content(item['retrieve_id'])
print(content)
```
## Hosted Content
We also host the content for 7 days, so you can retrieve it multiple times without re-scraping.
Example of a hosted url for [markdown content](https://olostep-storage.s3.us-east-1.amazonaws.com/markDown_p64bro0q3n_924d0fa125efcc5c.txt)
## Example Use Cases
### 1. Build Search Engines
Use Olostep to extract content from industry-specific websites (legal, medical, AI) and build a searchable database.
### 2. Website Monitoring
Monitor product availability, price changes, or news updates on multiple websites by scheduling daily batch scrapes.
### 3. Social Media Monitoring
Scrape mentions of your brand or keywords across forums or content sources and extract structured data.
### 4. Aggregators
Build a job board, news aggregator, or real estate listing platform by pulling data from dozens of sources.
## Conclusion
With batch scraping, you can extract content from up to 100k URLs quickly and efficiently. Whether you're building search tools, aggregators, or monitoring systems, Olostep Batches simplify the job.
Want to extract only structured data? Use [Parsers](https://docs.olostep.com/features/structured-content/parsers) to get just the fields you need. Need help? Reach out to `info@olostep.com` for support or have us write custom scripts for your use case.
# Crawl and Extract Content from Stripe's Blog Pages
Source: https://docs.olostep.com/examples/crawl-matching-pages
Learn how to crawl and extract content from Stripe's blog posts.
## Overview
This guide will show you how to:
* Start a crawl specifically targeting Stripe's blog posts
* Monitor the crawl progress
* Retrieve and process the crawled content
## Crawling Stripe's Blog Pages
To crawl Stripe's blog pages, use the crawls endpoint with pattern matching to target specific blog URLs. This will fetch the full HTML content of each page, which you can then process to extract the information you need.
```python crawl_stripe_blog.py theme={null}
import requests
import time
import json
from datetime import datetime
# Configuration
API_URL = 'https://api.olostep.com/v1'
API_KEY = ''
HEADERS = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {API_KEY}'
}
# Record start time for crawl duration tracking
crawl_start_time = time.time()
print(f"[{datetime.now().strftime('%H:%M:%S')}] Starting Stripe blog crawl...")
# Start a crawl focused on Stripe's engineering blog posts
# You can adjust the patterns based on your specific interests
payload = {
"start_url": "https://stripe.com/blog",
"include_urls": ["/blog/engineering/**"], # Focus on engineering posts
"max_pages": 25 # Limit to 25 pages for this example
}
# Start the crawl
print("Starting crawl of Stripe's engineering blog posts...")
response = requests.post(f'{API_URL}/crawls', headers=HEADERS, json=payload)
data = response.json()
crawl_id = data['id']
print(f"Crawl started with ID: {crawl_id}")
# Monitor crawl progress
while True:
status_response = requests.get(f'{API_URL}/crawls/{crawl_id}', headers=HEADERS)
status_data = status_response.json()
print(f"Crawl status: {status_data['status']} - Pages crawled: {status_data.get('pages_count', 0)}")
if status_data['status'] == 'completed' or status_data['status'] == 'failed':
break
# Wait 5 seconds before checking again
time.sleep(5)
# Calculate and display crawl duration
crawl_duration = time.time() - crawl_start_time
print(f"[{datetime.now().strftime('%H:%M:%S')}] Crawl completed in {crawl_duration:.2f} seconds")
```
## Converting Blog Content to Markdown
One powerful way to use the crawled content is to convert it to markdown format, which is ideal for feeding into LLMs or creating a knowledge base. Here's how to retrieve and convert the blog content to markdown:
```python blog_to_markdown.py theme={null}
import requests
import time
import json
from datetime import datetime
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
# Configuration
API_URL = 'https://api.olostep.com/v1'
API_KEY = ''
HEADERS = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {API_KEY}'
}
# Function to retrieve content with markdown format
def retrieve_content(retrieve_id, formats):
params = {
"retrieve_id": retrieve_id,
"formats": json.dumps(formats)
}
response = requests.get(f"{API_URL}/retrieve", headers=HEADERS, params=params)
return response.json()
# Continuing from the previous crawl example
if status_data['status'] == 'completed':
print(f"\nCrawl completed! Retrieved {status_data['pages_count']} pages.")
pages_response = requests.get(f'{API_URL}/crawls/{crawl_id}/pages', headers=HEADERS)
pages_data = pages_response.json()
# Create output directory if it doesn't exist
os.makedirs("output", exist_ok=True)
# Prepare to collect markdown content
markdown_pages = []
total_pages = len(pages_data['pages'])
# Process pages in parallel to get markdown content
with ThreadPoolExecutor(max_workers=10) as executor:
# Create futures for content retrieval
future_to_page = {
executor.submit(retrieve_content, page['retrieve_id'], ["markdown"]): page
for page in pages_data['pages']
}
# Process results as they complete
for i, future in enumerate(as_completed(future_to_page), 1):
page = future_to_page[future]
url = page['url']
print(f"Processing {i}/{total_pages}: {url}")
try:
content_data = future.result()
if content_data and "markdown_content" in content_data:
markdown_pages.append({
'url': url,
'title': page['title'],
'markdown_content': content_data['markdown_content']
})
print(f"✓ Markdown content retrieved for {url}")
else:
print(f"⚠ No markdown content for {url}")
except Exception as e:
print(f"❌ Error retrieving content for {url}: {str(e)}")
# Save all markdown content to a single file
output_file = "output/stripe_blog_markdown.md"
with open(output_file, "w", encoding="utf-8") as f:
for page in markdown_pages:
# Write page header with title and URL
f.write(f"URL: {page['url']}\n\n")
# Write the markdown content
f.write(f"{page['markdown_content']}\n\n")
# Add separator between pages
f.write("---\n\n")
print(f"✓ Added markdown content from {page['url']}")
print(f"\n✅ Process complete! All markdown content has been saved to '{output_file}'")
print(f"Total pages processed: {len(markdown_pages)}")
else:
print(f"Crawl failed with status: {status_data['status']}")
```
### Example Markdown Output
The resulting markdown file will contain all the crawled blog content in a clean, structured format:
```markdown theme={null}
URL: https://stripe.com/blog/using-ml-to-detect-and-respond-to-performance-degradations
## Using ML to detect and respond to performance degradations
By Jane Smith, Senior Engineer at Stripe
At Stripe, we process millions of API requests every day...
---
URL: https://stripe.com/blog/building-robust-payment-systems
## Building a robust payment system
By John Doe, Engineering Manager
Reliability is at the core of Stripe's infrastructure...
---
```
## Next Steps
Now that you've successfully crawled and extracted content from Stripe's blog, you can:
1. **Expand your crawl**: Modify the `include_urls` parameter to crawl other sections of Stripe's blog
2. **Implement regular updates**: Set up a scheduled job to periodically crawl for new content
3. **Perform deeper analysis**: Use NLP tools to extract insights from the blog content
4. **Build a search engine**: Create a searchable database of Stripe's blog content
5. **Feed into LLMs**: Use the markdown content as context for LLMs to answer questions about Stripe's engineering practices
Using Olostep's content crawling capabilities, you can build powerful tools for monitoring and analyzing any website's content strategy.
# Extract Blog URLs from Stripe's Website
Source: https://docs.olostep.com/examples/extract-specific-paths
Filter and extract only the blog URLs from Stripe's website for targeted content analysis.
## Overview
Instead of [mapping entire website](/examples/get-website-structure), you might want to focus on specific sections. In this guide, we'll show you how to extract only the blog URLs from Stripe's website.
## Extracting Only Blog URLs
To extract only blog URLs from Stripe's website, use the maps endpoint with path pattern filters. The `include_urls` parameter allows you to specify exactly which URL patterns you want to include in the results.
```python extract_stripe_blog_urls.py theme={null}
import requests
import time
import json
# Configuration
API_URL = 'https://api.olostep.com/v1'
API_KEY = ''
HEADERS = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {API_KEY}'
}
# Start time for latency tracking
start_time = time.time()
# Define the payload with URL patterns to include
payload = {
"url": "https://stripe.com",
"include_urls": ["/blog", "/blog/**"] # Match /blog and all paths under /blog
}
# Make the request
response = requests.post(f'{API_URL}/maps', headers=HEADERS, json=payload)
# Calculate latency
latency = round((time.time() - start_time) * 1000, 2)
print(f"Request completed in {latency}ms")
# Process the results
data = response.json()
print(f"Found {data['urls_count']} blog URLs on Stripe's website")
# Print the first 10 URLs as a sample
print("\nSample blog URLs:")
for url in data['urls'][:10]:
print(f"- {url}")
# Save blog URLs to a file for further processing
with open('stripe_blog_urls.json', 'w') as f:
json.dump(data, f, indent=2)
print(f"\nAll blog URLs saved to stripe_blog_urls.json")
```
### Understanding the URL Patterns
In the example above, we're using two pattern specifications:
* `/blog` - Matches exactly the main blog page ([https://stripe.com/blog](https://stripe.com/blog))
* `/blog/**` - Matches all subpaths under /blog, including individual blog posts, category pages, etc.
This combination ensures we capture all blog-related content while excluding other sections of the website.
### Example Response
```json theme={null}
{
"id": "map_xyz789abc",
"urls_count": 278,
"urls": [
"https://stripe.com/blog",
"https://stripe.com/blog/page/1",
"https://stripe.com/blog/page/2",
"https://stripe.com/blog/engineering",
"https://stripe.com/blog/product",
"https://stripe.com/blog/how-we-built-it-usage-based-billing",
"https://stripe.com/blog/using-ml-to-detect-and-respond-to-performance-degradations",
"https://stripe.com/blog/stripe-radar-responded-to-card-testing",
"https://stripe.com/blog/future-of-real-time-payments",
"https://stripe.com/blog/ml-flywheel-improve-models"
// ... additional URLs omitted for brevity
]
}
```
## Filtering Blog URLs by Category
You can further refine your extraction to focus on specific blog categories. For example, if you're only interested in Stripe's engineering blog posts:
```python extract_engineering_blog_urls.py theme={null}
# Define the payload with more specific URL patterns
payload = {
"url": "https://stripe.com",
"include_urls": ["/blog/engineering", "/blog/engineering/**"]
}
```
## Next Steps
Now that you have extracted all of Stripe's blog URLs,
1. You can fetch their content individually using the [scrape API](../../api-reference/scrapes/create).
2. Or, use the [next guide](/examples/crawl-matching-pages) to crawl and extract the actual content from these blog pages directly with inbuilt filters.
# Fetch data from popular AI search tools
Source: https://docs.olostep.com/examples/geo
Extract structured data from AI-powered search engines using batch requests
## Overview
This guide demonstrates how to use Olostep's Batches endpoint to fetch data from popular AI search tools including Google AI Mode, Gemini, Microsoft Copilot, ChatGPT, Perplexity, and Google AI Overview.
By leveraging specialized parsers for each platform, you can extract structured search results at scale, making it ideal for:
* Competitive intelligence gathering
* Multi-platform search result comparison
* AI search engine monitoring
## Request Formulation
### Step 1: Prepare Your Queries
Create an array of queries you want to search. For this demo, we'll query news across different cities:
```javascript theme={null}
const axios = require('axios');
// Configuration
const CONFIG = {
url: 'https://api.olostep.com/v1/batches',
token: 'YOUR_API_KEY_HERE'
};
// Demo: Generate queries for different cities
const BASE_QUERY = 'what is the news today in';
const CITIES = [
'New York', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix',
'Philadelphia', 'San Antonio', 'San Diego', 'Dallas', 'San Jose'
];
```
### Step 2: Generate Items for Each AI Tool
Each AI search tool requires a specific parser ID and URL structure. Below are the item generation functions for each platform:
**Parser ID:** `@olostep/google-aimode-results`
```javascript theme={null}
const generateAIModeItems = () => {
return CITIES.map((city, index) => {
const query = `${BASE_QUERY} ${city}`;
const encodedQuery = encodeURIComponent(query);
return {
url: `https://google.com/aimode?q=${encodedQuery}`,
custom_id: (index + 1).toString()
};
});
};
```
**Parser ID:** `@olostep/chatgpt-results`
```javascript theme={null}
const generateChatGPTItems = () => {
return CITIES.map((city, index) => {
const query = `${BASE_QUERY} ${city}`;
const encodedQuery = encodeURIComponent(query);
return {
url: `https://chatgpt.com/?q=${encodedQuery}`,
custom_id: (index + 1).toString()
};
});
};
```
**Parser ID:** `@olostep/perplexity-results`
```javascript theme={null}
const generatePerplexityItems = () => {
return CITIES.map((city, index) => {
const query = `${BASE_QUERY} ${city}`;
const encodedQuery = encodeURIComponent(query);
return {
url: `https://www.perplexity.ai/?q=${encodedQuery}`,
custom_id: (index + 1).toString()
};
});
};
```
**Parser ID:** `@olostep/google-ai-overview-results`
```javascript theme={null}
const generateGoogleAIOverviewItems = () => {
return CITIES.map((city, index) => {
const query = `${BASE_QUERY} ${city}`;
const encodedQuery = encodeURIComponent(query);
return {
url: `https://www.google.com/search?q=${encodedQuery}`,
custom_id: (index + 1).toString()
};
});
};
```
**Parser ID:** `@olostep/gemini-results`
```javascript theme={null}
const generateGeminiItems = () => {
return CITIES.map((city, index) => {
const query = `${BASE_QUERY} ${city}`;
const encodedQuery = encodeURIComponent(query);
return {
url: `https://gemini.google.com/?q=${encodedQuery}`,
custom_id: (index + 1).toString()
};
});
};
```
**Parser ID:** `@olostep/microsoft-copilot-results`
```javascript theme={null}
const generateCopilotItems = () => {
return CITIES.map((city, index) => {
const query = `${BASE_QUERY} ${city}`;
const encodedQuery = encodeURIComponent(query);
return {
url: `https://copilot.microsoft.com/chats?q=${encodedQuery}`,
custom_id: (index + 1).toString()
};
});
};
```
### Step 3: Submit the Batch Request
Submit your batch request with the generated items:
```javascript theme={null}
const response = await axios.post(CONFIG.url, {
parser: { id: '@olostep/gemini-results' },
items: generateGeminiItems()
}, {
headers: {
'Authorization': `Bearer ${CONFIG.token}`,
'Content-Type': 'application/json'
}
});
const batchId = response.data.id;
```
**Note:** After submitting, [poll for completion](/examples/batch#step-2-monitor-batch-status) to retrieve results or listen for [webhook events](/api-reference/common/webhooks).
## Response Format
After submitting a batch request and polling for completion, you'll receive responses in the following format:
```json theme={null}
{
"url": "https://www.google.com//search?q=what+is+the+news+today+in+Austin&hl=en&udm=50&aep=11&newwindow=1&sei=mt_oaPvDBrKh5NoPy9W1sQE&mstk=AUtExfANCngr4KIDEH7t1EJsJ3xHfdsjka647_hz7r0UJWh1VM4FhWV9j1f2QOy0ylJU2l9-zWCxfORo5WzWeAN52_oVMM7nGAgEIRdyzsjtT7h1qhBn8Qj2RiN8HFQke6uYjmqnTeR4O1opgHbiLdAe5ZNfkzDyE_9O2zE&csuir=1",
"prompt": "what is the news today in Austin",
"answer_markdown": "In Austin news, officials announced that progress has been made on the city's homelessness response \n\n. \n\n**Top story: Homelessness** \n\n* Austin officials and local advocates reported \"real, measurable progress\" in addressing the needs of the city's homeless population.\n* The announcement came ahead of a report from the Ending Community Homelessness Coalition (ECHO)... ",
"sources": [
{
"url": "https://www.kxan.com/video/austin-mayor-citys-decreased-homelessness-is-a-big-deal/11151402/#:~:text=Elected%20city%20and%20county%20officials%2C%20along%20with,experiencing%20homelessness%20in%20Austin.%20Read%20More:%20https://www.kxan.com/news/local/austin/echo%2Dto%2Dpresent%2Dreport%2Don%2Dstate%2Dof%2Daustins%2Dhomelessness%2Dresponse%2Dsystem/",
"title": "KXAN\n·",
"description": "Austin mayor: City's decreased homelessness is 'a big deal'",
"icon": null,
"domain": "https://www.kxan.com",
"cited": true
},
{
"url": "https://www.fox7austin.com/tag/us/tx/travis-county/austin/east-austin#:~:text=Austin%20pd%20arrests%20man%20in%20deadly%20east%20austin%20double%20shooting",
"title": "FOX 7 Austin",
"description": "East Austin",
"icon": null,
"domain": "https://www.fox7austin.com",
"cited": true
},
...
],
"country": null
}
```
```json theme={null}
{
"url": "https://chatgpt.com/?model=auto&q=what%20is%20the%20news%20today%20in%20Austin",
"prompt": "what is the news today in Austin",
"answer_markdown": "Here's a roundup of **today's top news in Austin, Texas**:\n\n---\n\n## 🏛️ Local Government\n\n* **New Homeless Navigation Center Approved:** The Austin City Council approved the purchase of a commercial property on South Interstate 35...\n\n[Council OKs purchase of site - Austin Monitor](https://austinmonitor.com/stories/2025/10/council-oks-purchase-of-site-for-new-homeless-navigation-center/)\n\n## 📚 Education\n\n* **Austin ISD Consolidation:** Families rally at Austin ISD headquarters to oppose consolidation plan...\n\n[Austin's Leading Local News - KVUE](https://www.kvue.com/)",
"inline_references": [
{
"url": "https://austinmonitor.com/stories/2025/10/council-oks-purchase-of-site-for-new-homeless-navigation-center/",
"text": "Council OKs purchase of site - Austin Monitor",
"position": 1
},
{
"url": "https://www.kvue.com/",
"text": "Austin's Leading Local News - KVUE",
"position": 2
},
...
],
"sources": [
{
"url": "https://austinmonitor.com/stories/2025/10/council-oks-purchase-of-site-for-new-homeless-navigation-center/",
"title": "Council OKs purchase of site for new homeless navigation center - Austin Monitor",
"snippet": "City Council voted 8–3 on Thursday to purchase a commercial property on South Interstate 35...",
"cited": true,
"date_published": "2025-10-10T12:00:00.000Z",
"attribution": "austinmonitor.com"
},
{
"url": "https://www.kvue.com/",
"title": "Austin's Leading Local News - KVUE",
"snippet": "Families rally at Austin ISD headquarters to oppose consolidation plan...",
"cited": true,
"date_published": null,
"attribution": "www.kvue.com"
},
...
],
"products": [],
"network_search_calls": {
"user_query": "what is the news today in Austin",
"default_model_slug": "auto",
"model_slug": "gpt-5-2",
"search_triggered": true,
"search_queries": [
{
"query": "Austin Texas news today October 2025",
"type": "model_query"
},
...
],
"search_result_groups": [
{
"domain": "austinmonitor.com",
"entries": [
{
"url": "https://austinmonitor.com/stories/2025/10/council-oks-purchase-of-site-for-new-homeless-navigation-center/",
"title": "Council OKs purchase of site for new homeless navigation center",
"snippet": "City Council voted 8–3 on Thursday to purchase a commercial property...",
"pub_date": "2025-10-10T12:00:00.000Z",
"attribution": "austinmonitor.com",
"ref_type": "news",
"ref_index": 1
}
]
},
...
]
}
}
```
```json theme={null}
{
"url": "https://www.perplexity.ai/search/what-is-the-news-today-in-aust-_3zTmnq8QGiB7f2f.ESHgw",
"prompt": "what is the news today in Austin",
"answer_markdown": "\nToday's news in Austin includes both local and statewide developments. Here's a summary of the major headlines:\n\n- Local Government\n - New Homeless Navigation Center: The Austin City Council approved the purchase of a commercial property on South Interstate 35 to serve as the city's first housing navigation center. [austinmonitor](https://austinmonitor.com/stories/2025/10/council-oks-purchase-of-site-for-new-homeless-navigation-center/)\n - Coffee Shop Zoning: The Austin City Council is working to make it easier to open walkable neighborhood coffee shops.\n- Education\n - Austin ISD Consolidation: Community members are protesting the Austin Independent School District's plan to close 13 schools, with a possible TEA takeover looming. Families rally at Austin ISD headquarters to oppose the consolidation plan. [kvue](https://www.kvue.com/)\n- Crime & Safety\n - North Austin Homicide: Police have identified a woman found dead behind a business on Research Boulevard. A suspect has been arrested.\n - Airport Security: A suspicious package found at Austin's airport has since been cleared.\n\nFor the latest updates, check local news sources like KUT, KVUE, and the Austin Monitor.\n",
"sources": [
{
"position": 1,
"label": "KUT Homepage | KUT Radio, Austin's NPR Station",
"url": "https://www.kut.org/",
"description": ""
},
{
"position": 2,
"label": "Austin News, Weather, Sports, Breaking News - CBS Austin",
"url": "https://cbsaustin.com/",
"description": ""
},
{
"position": 3,
"label": "Council OKs purchase of site for new homeless navigation center",
"url": "https://austinmonitor.com/stories/2025/10/council-oks-purchase-of-site-for-new-homeless-navigation-center/",
"description": ""
},
{
"position": 4,
"label": "Austin's Leading Local News - KVUE",
"url": "https://www.kvue.com/",
"description": ""
},
...
],
"related_queries": [
"Show top local headlines in Austin right now",
"What are the latest Austin ISD school closure updates",
"Austin City Council meeting highlights today"
],
"shopping_cards": [],
"videos": [],
"images": [],
"hotels": [],
"places": [],
"search_model_queries": [
{
"query": "what is the news today in Austin",
"engine": "web",
"limit": 8
}
]
}
```
```json theme={null}
{
"url": "https://www.google.com/search?q=what%20is%20the%20news%20today%20in%20Austin&gl=IN&hl=en",
"prompt": "what is the news today in Austin",
"answer_markdown": "Today's top news in Austin includes the Texas House failing to meet a quorum due to a Democratic standoff, a suspect being arrested for a North Austin homicide, and a suspicious package found at the Austin airport that has since been cleared. Other local headlines involve community protests against proposed Austin ISD school closures, the City Council's effort to simplify opening neighborhood coffee shops, and a man's 80-year prison sentence for his fifth DWI...",
"sources": [
{
"title": "Austin News, Weather, Sports, Breaking News",
"url": "https://cbsaustin.com/#:~:text=%22I%20feel%20sad%2C%22%20Austin,Roberson%20in%20shaken%20baby%20case",
"index": 0
},
{
"title": "News - FOX 7 Austin",
"url": "https://www.fox7austin.com/news#:~:text=Concerned%20community%20members%20protest%20proposed,multiple%20break%2Dins%20in%20custody",
"index": 1
},
...
],
"text_blocks": [
{
"type": "paragraph",
"snippet": "Today's top news in Austin includes the Texas House failing to meet a quorum due to a Democratic standoff, a suspect being arrested for a North Austin homicide, and a suspicious package found at the Austin airport that has since been cleared. Other local headlines involve community protests against proposed Austin ISD school closures, the City Council's effort to simplify opening neighborhood coffee shops, and a man's 80-year prison sentence for his fifth DWI conviction.",
"snippet_highlighted_words": "the Texas House failing to meet a quorum due to a Democratic standoff, a suspect being arrested for a North Austin homicide, and a suspicious package found at the Austin airport that has since been cleared"
},
{
"type": "list",
"list": [
{
"type": "paragraph",
"snippet": "Democratic Standoff: The Texas House is unable to meet a quorum because Texas Democrats are continuing their boycott of the legislative session."
},
{
"type": "paragraph",
"snippet": "Proposed ISD Closures: Community members are protesting the Austin Independent School District's plan to close 13 schools, with a possible TEA takeover looming."
},
{
"type": "paragraph",
"snippet": "Coffee Shops: The Austin City Council is working to make it easier to open walkable neighborhood coffee shops."
},
{
"type": "paragraph",
"snippet": "Homeless Navigation Center: The city council has approved the purchase of a new property for a homeless navigation center."
}
],
"title": "Austin Politics & Governance"
},
...
]
}
```
```json theme={null}
{
"url": "https://gemini.google.com/",
"prompt": "what is the news today in Austin",
"answer_markdown": "Here are some of the top local news headlines in Austin, Texas:\n\n### Local Government and Community\n\n* **New Homeless Navigation Center Approved:** The Austin City Council approved the purchase of a commercial property on South Interstate 35...\n\n[Council OKs purchase of site for new homeless navigation center - Austin Monitor](https://austinmonitor.com/stories/2025/10/council-oks-purchase-of-site-for-new-homeless-navigation-center/)\n\n* **Austin ISD Consolidation:** Families rally at Austin ISD headquarters to oppose consolidation plan...\n\n[Austin's Leading Local News - KVUE](https://www.kvue.com/)",
"sources": [
{
"position": 1,
"label": "Council OKs purchase of site for new homeless navigation center - Austin Monitor",
"url": "https://austinmonitor.com/stories/2025/10/council-oks-purchase-of-site-for-new-homeless-navigation-center/",
"description": "City Council voted 8–3 on Thursday to purchase a commercial property on South Interstate 35 to serve as the city's first housing navigation center...",
"confidence_level": 0.9781935
},
{
"position": 2,
"label": "Austin's Leading Local News - KVUE",
"url": "https://www.kvue.com/",
"description": "Families rally at Austin ISD headquarters to oppose consolidation plan. Parents are urging Austin ISD leaders to...",
"confidence_level": 0.9623676
},
...
],
"links_attached": true,
"model": "gemini"
}
```
```json theme={null}
{
"url": "https://copilot.microsoft.com/chats/G7TZeFAA3dMces2Cb5J3N",
"prompt": "what is the news today in Austin",
"answer_markdown": "Here's a roundup of today's top stories from Austin:\n\n## 🕵️ Crime & Safety\n\n**Homicide Investigation in North Austin** Police have identified 43-year-old Mary Gonzales as the woman found dead behind a business on Research Boulevard. A suspect, 21-year-old, has been arrestedMSN News.\n\n**Suspicious Package at Airport** Police investigated after a suspicious package was found at Austin's airport, which has since been clearedMSN News.\n\n## 🏛️ Local Government\n\n**Homeless Navigation Center** The city council has approved the purchase of a new property for a homeless navigation centerAustin Monitor.\n\n[MSN NewsWoman found dead behind North Austin business identified; suspect arrested](https://www.msn.com/en-us/news/crime/woman-found-dead-behind-north-austin-business-identified-suspect-arrested/ar-AA1ObaMU)\n[MSN NewsPolice investigating after suspicious package found at Austin's airport](https://www.msn.com/en-us/news/crime/police-investigating-after-suspicious-package-found-at-austins-airport/ar-AA1Ob39x)",
"sources": [
{
"url": "https://www.msn.com/en-us/news/crime/woman-found-dead-behind-north-austin-business-identified-suspect-arrested/ar-AA1ObaMU",
"title": "Woman found dead behind North Austin business identified; suspect arrested",
"position": 319,
"icon_url": "https://services.bingapis.com/favicon?url=www.msn.com"
},
{
"url": "https://www.msn.com/en-us/news/crime/police-investigating-after-suspicious-package-found-at-austins-airport/ar-AA1Ob39x",
"title": "Police investigating after suspicious package found at Austin's airport",
"position": 555,
"icon_url": "https://services.bingapis.com/favicon?url=www.msn.com"
},
...
]
}
```
**Full response examples:**
* [Google AI Mode](https://olostep-storage.s3.us-east-1.amazonaws.com/json_wcez50jwt6_2.json)
* [ChatGPT](https://olostep-storage.s3.us-east-1.amazonaws.com/json_9q5mzct5fe_2.json)
* [Perplexity](https://olostep-storage.s3.us-east-1.amazonaws.com/json_4930ezsqmw_2.json)
* [Google AI Overview](https://olostep-storage.s3.us-east-1.amazonaws.com/json_tshp1kuuyn_2.json)
* [Gemini](https://olostep-storage.s3.us-east-1.amazonaws.com/json_y2tqdrfaro_2.json)
* [Microsoft Copilot](https://olostep-storage.s3.us-east-1.amazonaws.com/json_hyjx3dykha_1.json)
## Pricing and Geolocation Matrix
The following table outlines the support for country-specific search and the credits consumed by each parser.
| Feature | Google AI Mode | ChatGPT | Perplexity | Google AI Overview | Gemini | Microsoft Copilot |
| --------------------- | :------------: | :-----: | :--------: | :----------------: | :----: | :---------------: |
| **Country Supported** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **Batch Size Limit** | 2500 | 2500 | 2500 | 2500 | 2500 | 1000 |
| **Credits** | 3 | 5 | 3 | 3 | 3 | 3 |
### Checking Supported Countries
Use the following endpoint to retrieve the list of supported countries for each parser:
```
GET https://api.olostep.com/v1/countries?service=batches&parser={parser_id}
```
**Example Request:**
```bash theme={null}
curl "https://api.olostep.com/v1/countries?service=batches&parser=@olostep/perplexity-results"
```
Missing something? Reach out to `info@olostep.com` for support or custom implementation assistance.
# Get the Markdown of a Website
Source: https://docs.olostep.com/examples/get-markdown-from-website
Learn how to extract content as LLM-friendly markdown from any web page.
## Overview
Olostep's [scrape](../../features/scrapes) endpoint allows to extract content from any website. Content in markdown is useful if you want to feed it to an LLM without all the HTML.
In this guide we will see how to extract markdown from a website like `https://www.nea.com/team`.
## Prerequisites
Before getting started, ensure you have the following:
* A valid Olostep API key. You can get one by signing up at [Olostep](https://www.olostep.com/dashboard).
* Python installed on your system
* The `requests` and `json` libraries (these come pre-installed with Python, but you can install them using `pip install requests` if needed)
## Extracting Text from a Website
The following Python script demonstrates how to extract text and markdown content from a website using Olostep's API.
```python theme={null}
import requests
import json
url = "https://api.olostep.com/v1/scrapes"
payload = {
"url_to_scrape": "https://www.nea.com/team",
"country": "US",
"formats": ["markdown"],
"wait_before_scraping": 0,
"remove_css_selectors": "default",
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
response = requests.request("POST", url, json=payload, headers=headers)
print(json.dumps(response.json(), indent=4))
```
## Example Response
A successful response will look something like this:
```json theme={null}
{
"id": "scrape_63x2e5sf5r",
"object": "scrape",
"created": 1740341743,
"metadata": {},
"retrieve_id": "63x2e5sf5r",
"url_to_scrape": "https://www.nea.com/team",
"result": {
"html_content": null,
"markdown_content": "NEA ….",
"text_content": null,
"json_content": null,
"llm_extract": null,
"screenshot_hosted_url": null,
"html_hosted_url": null,
"markdown_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/markDown_63x2e5sf5r.txt",
"json_hosted_url": null,
"text_hosted_url": null,
"links_on_page": [],
"page_metadata": {
"status_code": 200,
"title": ""
}
}
}
```
## Explanation
* `url_to_scrape`: specifies the website URL to extract content from.
* `formats`: defines the output formats (text in this case).
* `Authorization`: contains your API key to authenticate the request.
* The response is formatted as JSON and printed for readability.
## Conclusion
Using Olostep, you can easily extract markdown content from any website. This is useful if you want to get content from a website and feed it to an LLM for data extraction and analysis. If you want to extract content at scale from the same website over and over (e.g. monitoring data, price tracking, etc...) we recommend using a [custom parser](../../features/structured-content/parsers) to get the content in JSON format.
# Extract All URLs from Stripe's Website
Source: https://docs.olostep.com/examples/get-website-structure
Get a complete map of Stripe's website structure to understand its organization and discover available content.
## Overview
Before diving into specific sections of a website, it's often useful to get a complete picture of its structure. In this guide, we'll show you how to extract all URLs from Stripe's website, which will help you:
* Understand the overall site architecture
* Discover content sections you might not be aware of
* Use LLMs to decide which URLs to further scrape
## Extracting All Stripe URLs
To extract all URLs from Stripe's website, use the maps endpoint with Stripe's domain. This will return a comprehensive list of all discoverable URLs on their site.
```python get_stripe_urls.py theme={null}
import requests
import time
import json
# Configuration
API_URL = 'https://api.olostep.com/v1'
API_KEY = ''
HEADERS = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {API_KEY}'
}
# Start time for latency tracking
start_time = time.time()
# Define the payload with just the base URL
payload = {
"url": "https://stripe.com"
}
# Make the request
response = requests.post(f'{API_URL}/maps', headers=HEADERS, json=payload)
# Calculate latency
latency = round((time.time() - start_time) * 1000, 2)
print(f"Request completed in {latency}ms")
# Process the results
data = response.json()
print(f"Found {data['urls_count']} URLs on Stripe's website")
# Print the first 10 URLs as a sample
print("\nSample URLs:")
for url in data['urls'][:10]:
print(f"- {url}")
# Save all URLs to a file for further analysis
with open('stripe_urls.json', 'w') as f:
json.dump(data, f, indent=2)
print(f"\nAll URLs saved to stripe_urls.json")
```
### Example Response
```json theme={null}
{
"id": "map_abc123xyz",
"urls_count": 3842,
"urls": [
"https://stripe.com",
"https://stripe.com/about",
"https://stripe.com/blog",
"https://stripe.com/docs",
"https://stripe.com/pricing",
"https://stripe.com/customers",
"https://stripe.com/partners",
"https://stripe.com/enterprise",
"https://stripe.com/payments",
"https://stripe.com/billing"
// ... thousands more URLs
]
}
```
## Analyzing Stripe's Website Structure
After extracting all URLs, you can analyze the structure to identify patterns. This is particularly useful for understanding how Stripe organizes their content. For example, you might notice these URL patterns:
* `/blog/**` - Blog posts and articles
* `/docs/**` - Documentation pages
* `/payments/**` - Payment product information
* `/billing/**` - Billing product information
In some cases, you only want to get URLs in a specific section of the website. For instance, all blog posts. You can use our inbuilt filter in the [next guide](/examples/extract-specific-paths).
# Get Google Maps Results
Source: https://docs.olostep.com/examples/google-maps
SERP API to scrape Google Maps location data using Olostep and a Python code snippet.
## Overview
Olostep's API allows you to extract structured data from Google Maps locations by using parsers. These parsers transform the raw HTML of Google Maps pages into clean, structured JSON data that's ready for analysis or integration into your applications.
## Integration Example
To get parsed JSON content from Google Maps results, you need to include `json` in the formats parameter and specify the name of the parser `@olostep/google-maps` in the parser object.
Here's how to retrieve Google Maps location data in a structured format:
```python theme={null}
import requests
import json
url = "https://api.olostep.com/v1/scrapes"
payload = {
"url_to_scrape": "https://www.google.com/maps/search/Coffee+on+the+Rocks/@40.3699261,-105.5321621,17z?hl=en",
"formats": ["json"],
"parser": {"id": "@olostep/google-maps"},
"remove_css_selectors": "none",
"wait_before_scraping": 4000
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
response = requests.request("POST", url, json=payload, headers=headers)
print(json.dumps(response.json(), indent=4))
```
## Response Format
When you make a request to the Olostep API with the Google Maps parser, you'll receive a JSON response like the example below:
```json theme={null}
{
"id": "scrape_y6jwelcpdl",
"object": "scrape",
"created": 1742697931,
"metadata": {},
"retrieve_id": "y6jwelcpdl",
"url_to_scrape": "https://www.google.com/maps/search/Coffee+on+the+Rocks/@40.3699261,-105.5321621,17z?hl=en",
"result": {
"html_content": null,
"markdown_content": null,
"text_content": null,
"json_content": "{\"searchString\":\"\",\"rank\":null,\"searchPageUrl\":null,\"searchPageLoadedUrl\":null,\"isAdvertisement\":false,\"title\":\"Coffee on the Rocks\",\"subTitle\":null,\"description\":null,\"price\":\"\u00b7$1\u201310\",\"categoryName\":\"Cafe\",\"address\":\"510 Moraine Ave, Estes Park, CO 80517\",\"neighborhood\":\"\",\"street\":\"510 Moraine Ave\",\"city\":\"Estes Park\",\"postalCode\":\"80517\",\"state\":\"CO\",\"countryCode\":\"US\",\"website\":\"coffeeontherocks.org\",\"phone\":\"(970) 909-4836\",\"phoneUnformatted\":\"+19709094836\",\"claimThisBusiness\":false,\"location\":{\"lat\":null,\"lng\":null},\"locatedIn\":null,\"plusCode\":\"9F9C+X5 Estes Park, Colorado\",\"menu\":\"https://www.coffeeontherocks.org/\",\"totalScore\":4.6,\"permanentlyClosed\":false,\"temporarilyClosed\":false,\"placeId\":\"\",\"categories\":[\"Cafe\"],\"fid\":\"\",\"cid\":\"\",\"reviewsCount\":1555,\"reviewsDistribution\":{\"oneStar\":42,\"twoStar\":38,\"threeStar\":70,\"fourStar\":244,\"fiveStar\":1161},\"imagesCount\":9,\"imageCategories\":[\"All\",\"Latest\",\"Videos\",\"Menu\",\"Food & drink\",\"Vibe\",\"Garden\",\"Breakfast sandwich\",\"Street View & 360\u00b0\"],\"scrapedAt\":\"2025-03-23T02:45:31.153Z\",\"reserveTableUrl\":null,\"googleFoodUrl\":null,\"hotelStars\":null,\"hotelDescription\":null,\"checkInDate\":null,\"checkOutDate\":null,\"similarHotelsNearby\":null,\"hotelReviewSummary\":null,\"hotelAds\":[],\"openingHours\":[{\"day\":\"Saturday\",\"hours\":\"7\u202fAM\u20134\u202fPM\"},{\"day\":\"Sunday\",\"hours\":\"7\u202fAM\u20134\u202fPM\"},{\"day\":\"Monday\",\"hours\":\"7\u202fAM\u20134\u202fPM\"},{\"day\":\"Tuesday\",\"hours\":\"7\u202fAM\u20134\u202fPM\"},{\"day\":\"Wednesday\",\"hours\":\"7\u202fAM\u20134\u202fPM\"},{\"day\":\"Thursday\",\"hours\":\"7\u202fAM\u20134\u202fPM\"},{\"day\":\"Friday\",\"hours\":\"7\u202fAM\u20134\u202fPM\"}],\"peopleAlsoSearch\":[{\"category\":\"People also search for\",\"title\":\"Mountain Home Cafe\",\"reviewsCount\":1916,\"totalScore\":4.5},{\"category\":\"People also search for\",\"title\":\"Bird's Nest\",\"reviewsCount\":485,\"totalScore\":4.4},{\"category\":\"People also search for\",\"title\":\"Raven's Roast Coffee Lounge\",\"reviewsCount\":423,\"totalScore\":4.7},{\"category\":\"People also search for\",\"title\":\"Kind Coffee\",\"reviewsCount\":1856,\"totalScore\":4.7},{\"category\":\"People also search for\",\"title\":\"Inkwell & Brew\",\"reviewsCount\":869,\"totalScore\":4.6},{\"category\":\"People also search for\",\"title\":\"Big Horn Restaurant\",\"reviewsCount\":2936,\"totalScore\":4.3},{\"category\":\"People also search for\",\"title\":\"Brunch & Co\",\"reviewsCount\":244,\"totalScore\":4.1},{\"category\":\"People also search for\",\"title\":\"The egg of estes\",\"reviewsCount\":2857,\"totalScore\":4.6},{\"category\":\"People also search for\",\"title\":\"Notchtop Bakery & Cafe\",\"reviewsCount\":2237,\"totalScore\":4.6},{\"category\":\"People also search for\",\"title\":\"Kissing Moose Cafe\",\"reviewsCount\":36,\"totalScore\":4.7},{\"category\":\"People also search for\",\"title\":\"Starbucks\",\"reviewsCount\":113,\"totalScore\":3.8},{\"category\":\"People also search for\",\"title\":\"Rustic Cafe\",\"reviewsCount\":12,\"totalScore\":4.5}],\"placesTags\":[],\"reviewsTags\":[{\"title\":\"river\",\"count\":75},{\"title\":\"duck pond\",\"count\":48},{\"title\":\"breakfast sandwiches\",\"count\":42},{\"title\":\"rocky mountain national park\",\"count\":30},{\"title\":\"feed\",\"count\":29},{\"title\":\"patio\",\"count\":28},{\"title\":\"avocado toast\",\"count\":17},{\"title\":\"elk\",\"count\":11},{\"title\":\"creek\",\"count\":10},{\"title\":\"ducks and geese\",\"count\":9}],\"additionalInfo\":{\"Service options\":[{\"Dine-in\":true},{\"Takeout\":true},{\"Delivery\":true}],\"Popular for\":[{\"Lunch\":true},{\"Solo dining\":true},{\"Lunch\":true},{\"Dinner\":false}],\"Accessibility\":[{\"Wheelchair accessible entrance\":true}],\"Offerings\":[{\"Alcohol\":true},{\"Comfort food\":true},{\"Healthy options\":true},{\"Quick bite\":true},{\"Wine\":true},{\"Small plates\":false}],\"Dining options\":[{\"Lunch\":true},{\"Lunch\":true},{\"Dessert\":true},{\"Seating\":true},{\"Dinner\":false}],\"Amenities\":[{\"Restroom\":true}],\"Atmosphere\":[{\"Casual\":true}],\"Planning\":[{\"Accepts reservations\":false}],\"Payments\":[{\"Credit cards\":true},{\"Debit cards\":true},{\"NFC mobile payments\":true}],\"Children\":[{\"Good for kids\":true}]},\"gasPrices\":[],\"questionsAndAnswers\":[],\"updatesFromCustomers\":null,\"ownerUpdates\":[],\"url\":\"\",\"imageUrl\":\"https://lh5.googleusercontent.com/p/AF1QipOTzB5ux9tVMq4AOp4kCAsvwXsNA4LycnKFnMw3=w408-h305-k-no\",\"kgmid\":\"\",\"webResults\":[],\"parentPlaceUrl\":null,\"tableReservationLinks\":[],\"bookingLinks\":[],\"orderBy\":[{\"name\":\"coffeeontherocks.org\",\"orderUrl\":\"coffeeontherocks.org\"}],\"images\":[{\"imageUrl\":\"https://lh5.googleusercontent.com/p/AF1QipOTzB5ux9tVMq4AOp4kCAsvwXsNA4LycnKFnMw3=w408-h305-k-no\",\"authorName\":\"\",\"authorUrl\":\"\",\"uploadedAt\":\"\"}],\"imageUrls\":[\"https://lh5.googleusercontent.com/p/AF1QipOTzB5ux9tVMq4AOp4kCAsvwXsNA4LycnKFnMw3=w408-h305-k-no\"],\"reviews\":[{\"name\":\"Katie Steinbach\",\"text\":\"Such a fun place to try while you are in Estes! The coffee itself wasn\u2019t my favorite but the atmosphere and service made up for it. The staff was very friendly and we got our food and drinks within minutes of walking through the door. ...\",\"textTranslated\":null,\"publishAt\":\"6 months ago\",\"publishedAtDate\":\"2024-09-23T02:45:31.159Z\",\"likesCount\":1,\"reviewId\":\"ChRDSUhNMG9nS0VJQ0FnSURIc0pCShAB\",\"reviewUrl\":\"\",\"reviewerId\":\"110511108367107737234\",\"reviewerUrl\":\"https://www.google.com/maps/contrib/110511108367107737234/reviews?hl=en-US\",\"reviewerPhotoUrl\":\"https://lh3.googleusercontent.com/a-/ALV-UjVDOx8QHSyHNYBimKQAy2b185lLtLhNd-uhwGuh6AbuqUvkPH65=w36-h36-p-rp-mo-ba4-br100\",\"reviewerNumberOfReviews\":0,\"isLocalGuide\":true,\"reviewOrigin\":\"Google\",\"stars\":5,\"rating\":null,\"responseFromOwnerDate\":null,\"responseFromOwnerText\":null,\"reviewImageUrls\":[\"https://lh3.googleusercontent.com/geougc-cs/AIHozJK4iaCdo4hQkehYZn-i8dt1B4ZjvtpD453dVz_gqYNLzdWpJ-ITv2aXJnfXiTm4GgnB7lY_sUfpDWJSONdCiJS6IhwNvmognPjbKF_panMyZK-Kz1CVUUObomtxnxjtRa_s6IKGFA=w300-h225-p\",\"https://lh3.googleusercontent.com/geougc-cs/AIHozJLG7c8vxcShHisTgOsEe-hP9jbkTn7FcdoXXAOzw6GKfykwEeMjTYEtm1D0Ij-aNaJ10Qh3P7EcQiOT4SuGFlgBYm7rrBTSARt1P8gMbtFuwHyaqgIsnYWeP_Ksm-zD_8wwlC_a=w300-h225-p\",\"https://lh3.googleusercontent.com/geougc-cs/AIHozJLo94qIiyPmUorcS_YJuJXa1cgQaQwbdIewgZpsqbF-e4e6Vaa1DdOj3wD8Zdh9EeH1tPYCMd4_BTqM0eIgSQ-wPsrNgjvCJ8rPGBFFBaxqo_a9X4qzcx9d1-OiFDz_cDpIrTjmjA=w300-h225-p\",\"https://lh3.googleusercontent.com/geougc-cs/AIHozJIFNsk3L0IKaIcL_YsxGUPOZXIzn7iRohFepONb-6nCT72EQ2Tbxx5McohqMxP-xOUV_Pg-PB5k9zfhRT8cSvnZlMhUcRYKOPvUSjaToVhk4jUER-eRr8abU6GjVD4ek_7t8ZA=w300-h225-p\"],\"reviewContext\":{},\"reviewDetailedRating\":{}},{\"name\":\"Aaron Decker\",\"text\":\"What a wonderful coffee shop! Nice atmosphere, ducks!, delicious breakfast sandwich, and of course amazing coffee to hit the spot! ...\",\"textTranslated\":null,\"publishAt\":\"2 months ago\",\"publishedAtDate\":\"2025-01-23T03:45:31.159Z\",\"likesCount\":0,\"reviewId\":\"ChdDSUhNMG9nS0VJQ0FnSUNmZ1oydHpnRRAB\",\"reviewUrl\":\"\",\"reviewerId\":\"100124563986259402322\",\"reviewerUrl\":\"https://www.google.com/maps/contrib/100124563986259402322/reviews?hl=en-US\",\"reviewerPhotoUrl\":\"https://lh3.googleusercontent.com/a-/ALV-UjUWve1upybVvSo9jDd3PgnICSzEzWA_a-2qe42CwF39cRQhSxk0kA=w36-h36-p-rp-mo-ba8-br100\",\"reviewerNumberOfReviews\":0,\"isLocalGuide\":true,\"reviewOrigin\":\"Google\",\"stars\":5,\"rating\":null,\"responseFromOwnerDate\":null,\"responseFromOwnerText\":null,\"reviewImageUrls\":[\"https://lh3.googleusercontent.com/geougc-cs/AIHozJKr8SDZO4j-Wjzf39-2FMZ2YLZicF_vwNCVeFGbUNuMV2KQNbeZq-WxlwcAiTzx4ue_6UPLdEdqf91PqgYDYg5V1jqHzQSu1wy1Zhi-ardEnpFYpUjPldDAhqVqRQ6t39KzdiqM=w300-h225-p\",\"https://lh3.googleusercontent.com/geougc-cs/AIHozJIWUwX2P-cF8BHT87qgt5hqkXP9t2oc20xW7INb086bsp43uFyWRRjZeEuMQjaxK-lg7t7YYM7bbjoRVGuZhXfSTwm4c7nvuzvpq1Y6HZX9rc9YChEv9RARvXhCrTQvjb40n0-22Q=w300-h225-p\",\"https://lh3.googleusercontent.com/geougc-cs/AIHozJInSKZ9ZYNxvUAuQe8wNnGif7uottZ2-qE03vFnkLXrnHtQ0NxcyhgnCuFNbcaLwVLyTO-CmrQsXntHFMzbJtRNtENAcSfyIVskkiIF9gxNnRzSF2NvIEMJw2nU3W510G_JjC5g=w300-h225-p\",\"https://lh3.googleusercontent.com/geougc-cs/AIHozJKhWuvZR4xvgzDsPO56SS2_IC79HzVlKNvb4Sy9Phow5ugIhQRaffOkh70B4nW1nzjTtWgeCu5Gk8Z-MH6KxFOG0TqovuuZYshQU1qTvZPbon7e6gsWc3-mag9thRG5ujELQ5mEiQ=w300-h225-p\"],\"reviewContext\":{},\"reviewDetailedRating\":{}},{\"name\":\"Stephanie D\",\"text\":\"Stopped in around 9am on a Saturday. They were a bit busy but they were able to move the line quickly and there is plenty to look at while you wait. Our coffee was delicious but our croissant was even better. We got a bacon, egg and cheese ...\",\"textTranslated\":null,\"publishAt\":\"5 months ago\",\"publishedAtDate\":\"2024-10-23T02:45:31.159Z\",\"likesCount\":1,\"reviewId\":\"ChZDSUhNMG9nS0VJQ0FnSUNYbXFpbUJ3EAE\",\"reviewUrl\":\"\",\"reviewerId\":\"100254141532590923077\",\"reviewerUrl\":\"https://www.google.com/maps/contrib/100254141532590923077/reviews?hl=en-US\",\"reviewerPhotoUrl\":\"https://lh3.googleusercontent.com/a-/ALV-UjVC1QNXDVsl2RyOM9UNQelqeF3pFdyTb0RvviZmA7_3n5NTRrt3=w36-h36-p-rp-mo-ba4-br100\",\"reviewerNumberOfReviews\":0,\"isLocalGuide\":true,\"reviewOrigin\":\"Google\",\"stars\":4,\"rating\":null,\"responseFromOwnerDate\":null,\"responseFromOwnerText\":null,\"reviewImageUrls\":[\"https://lh3.googleusercontent.com/geougc-cs/AIHozJJct2N0-nz5ADJcFJ6zoT80k7xaSgq1MHl8AqO31k9-D_a6x90qciLdCni8vUQkwOO4iUIn_Bn5wiG1FecOafyRESitBfnxENj_pGdqRNroR9VvFbLXhRwLqMiXyIogWGZazx0=w300-h450-p\",\"https://lh3.googleusercontent.com/geougc-cs/AIHozJJoGd5NaZWjBrDCaov1klIkKsCOFfHuEAMQwgMqzTOBN9SRkMNKSO7b6TW9jxFWeYjJsFkf9GBm2rJy0BTOqibTRBH-CQ1CA8e-Bm9wMrUnjNFiLUngrj40c_4F78RxS1IQugK0xQ=w300-h225-p\",\"https://lh3.googleusercontent.com/geougc-cs/AIHozJJ2fQ7Idy4PX4fDSwjtrCub0-Oo0EL5G9Qsh1tfCldS6kMsU_pRTvqI4EegsoFPP-HOU3oWgAvZXXL9Zv1lGEzL_qA2yJezSr2kFtx7DFHj1-Tfn8P6UucOGq_QdHBslISiGTDy=w300-h225-p\"],\"reviewContext\":{},\"reviewDetailedRating\":{}}],\"userPlaceNote\":null,\"restaurantData\":{}}",
"llm_extract": null,
"screenshot_hosted_url": null,
"html_hosted_url": null,
"markdown_hosted_url": null,
"json_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json_y6jwelcpdl.json",
"text_hosted_url": null,
"links_on_page": [],
"page_metadata": {
"status_code": 200,
"title": ""
}
}
}
```
The response contains:
* **Basic request information**: `id`, `object`, `created` timestamp, `url_to_scrape`
* **Result object** with URLs to access different formats of the data
* **json\_content** with structured search results including:
* `Business details` (name, address, contact information)
* `Rating and review information`
* `Opening hours`
* `User reviews with text and images`
* `Similar places nearby`
* `Popular tags and amenities`
## Structured Response: json\_content
The json\_content field contains detailed information about the location. Here's an example of data you can expect:
```json theme={null}
{
"searchString": "",
"rank": null,
"searchPageUrl": null,
"searchPageLoadedUrl": null,
"isAdvertisement": false,
"title": "Coffee on the Rocks",
"subTitle": null,
"description": null,
"price": "·$1–10",
"categoryName": "Cafe",
"address": "510 Moraine Ave, Estes Park, CO 80517",
"neighborhood": "",
"street": "510 Moraine Ave",
"city": "Estes Park",
"postalCode": "80517",
"state": "CO",
"countryCode": "US",
"website": "coffeeontherocks.org",
"phone": "(970) 909-4836",
"phoneUnformatted": "+19709094836",
"claimThisBusiness": false,
"location": {
"lat": null,
"lng": null
},
"locatedIn": null,
"plusCode": "9F9C+X5 Estes Park, Colorado",
"menu": "https://www.coffeeontherocks.org/",
"totalScore": 4.6,
"permanentlyClosed": false,
"temporarilyClosed": false,
"placeId": "",
"categories": [
"Cafe"
],
"fid": "",
"cid": "",
"reviewsCount": 1555,
"reviewsDistribution": {
"oneStar": 42,
"twoStar": 38,
"threeStar": 70,
"fourStar": 244,
"fiveStar": 1161
},
"imagesCount": 10,
"imageCategories": [
"All",
"Latest",
"Videos",
"Menu",
"Food & drink",
"Vibe",
"Garden",
"Mountain",
"Breakfast sandwich",
"Street View & 360°"
],
"scrapedAt": "2025-03-22T01:18:42.728Z",
"reserveTableUrl": null,
"googleFoodUrl": null,
"hotelStars": null,
"hotelDescription": null,
"checkInDate": null,
"checkOutDate": null,
"similarHotelsNearby": null,
"hotelReviewSummary": null,
"hotelAds": [],
"openingHours": [
{
"day": "Friday",
"hours": "7 AM–4 PM"
},
{
"day": "Saturday",
"hours": "7 AM–4 PM"
},
{
"day": "Sunday",
"hours": "7 AM–4 PM"
},
{
"day": "Monday",
"hours": "7 AM–4 PM"
},
{
"day": "Tuesday",
"hours": "7 AM–4 PM"
},
{
"day": "Wednesday",
"hours": "7 AM–4 PM"
},
{
"day": "Thursday",
"hours": "7 AM–4 PM"
}
],
"peopleAlsoSearch": [
{
"category": "People also search for",
"title": "Mountain Home Cafe",
"reviewsCount": 1915,
"totalScore": 4.5
},
{
"category": "People also search for",
"title": "Bird's Nest",
"reviewsCount": 484,
"totalScore": 4.4
},
{
"category": "People also search for",
"title": "Raven's Roast Coffee Lounge",
"reviewsCount": 423,
"totalScore": 4.7
},
{
"category": "People also search for",
"title": "Kind Coffee",
"reviewsCount": 1856,
"totalScore": 4.7
},
{
"category": "People also search for",
"title": "Inkwell & Brew",
"reviewsCount": 869,
"totalScore": 4.6
},
{
"category": "People also search for",
"title": "Big Horn Restaurant",
"reviewsCount": 2936,
"totalScore": 4.3
},
{
"category": "People also search for",
"title": "Brunch & Co",
"reviewsCount": 244,
"totalScore": 4.1
},
{
"category": "People also search for",
"title": "The egg of estes",
"reviewsCount": 2854,
"totalScore": 4.6
},
{
"category": "People also search for",
"title": "Notchtop Bakery & Cafe",
"reviewsCount": 2237,
"totalScore": 4.6
},
{
"category": "People also search for",
"title": "Kissing Moose Cafe",
"reviewsCount": 36,
"totalScore": 4.7
},
{
"category": "People also search for",
"title": "Starbucks",
"reviewsCount": 113,
"totalScore": 3.8
},
{
"category": "People also search for",
"title": "Rustic Cafe",
"reviewsCount": 12,
"totalScore": 4.5
}
],
"placesTags": [],
"reviewsTags": [
{
"title": "river",
"count": 75
},
{
"title": "duck pond",
"count": 48
},
{
"title": "breakfast sandwiches",
"count": 41
},
{
"title": "rocky mountain national park",
"count": 30
},
{
"title": "feed",
"count": 30
},
{
"title": "patio",
"count": 28
},
{
"title": "avocado toast",
"count": 17
},
{
"title": "elk",
"count": 11
},
{
"title": "creek",
"count": 10
},
{
"title": "ducks and geese",
"count": 9
}
],
"additionalInfo": {
"Service options": [
{
"Dine-in": true
},
{
"Takeout": true
},
{
"Delivery": true
}
],
"Popular for": [
{
"Lunch": true
},
{
"Solo dining": true
},
{
"Lunch": true
},
{
"Dinner": false
}
],
"Accessibility": [
{
"Wheelchair accessible entrance": true
}
],
"Offerings": [
{
"Alcohol": true
},
{
"Comfort food": true
},
{
"Healthy options": true
},
{
"Quick bite": true
},
{
"Wine": true
},
{
"Small plates": false
}
],
"Dining options": [
{
"Lunch": true
},
{
"Lunch": true
},
{
"Dessert": true
},
{
"Seating": true
},
{
"Dinner": false
}
],
"Amenities": [
{
"Restroom": true
}
],
"Atmosphere": [
{
"Casual": true
}
],
"Planning": [
{
"Accepts reservations": false
}
],
"Payments": [
{
"Credit cards": true
},
{
"Debit cards": true
},
{
"NFC mobile payments": true
}
],
"Children": [
{
"Good for kids": true
}
]
},
"gasPrices": [],
"questionsAndAnswers": [],
"updatesFromCustomers": null,
"ownerUpdates": [],
"url": "",
"imageUrl": "https://lh5.googleusercontent.com/p/AF1QipOTzB5ux9tVMq4AOp4kCAsvwXsNA4LycnKFnMw3=w408-h305-k-no",
"kgmid": "",
"webResults": [],
"parentPlaceUrl": null,
"tableReservationLinks": [],
"bookingLinks": [],
"orderBy": [
{
"name": "coffeeontherocks.org",
"orderUrl": "coffeeontherocks.org"
}
],
"images": [
{
"imageUrl": "https://lh5.googleusercontent.com/p/AF1QipOTzB5ux9tVMq4AOp4kCAsvwXsNA4LycnKFnMw3=w408-h305-k-no",
"authorName": "",
"authorUrl": "",
"uploadedAt": ""
}
],
"imageUrls": [
"https://lh5.googleusercontent.com/p/AF1QipOTzB5ux9tVMq4AOp4kCAsvwXsNA4LycnKFnMw3=w408-h305-k-no"
],
"reviews": [
{
"name": "Katie Steinbach",
"text": "Such a fun place to try while you are in Estes! The coffee itself wasn’t my favorite but the atmosphere and service made up for it. The staff was very friendly and we got our food and drinks within minutes of walking through the door. ...",
"textTranslated": null,
"publishAt": "6 months ago",
"publishedAtDate": "2024-09-22T01:18:42.734Z",
"likesCount": 1,
"reviewId": "ChRDSUhNMG9nS0VJQ0FnSURIc0pCShAB",
"reviewUrl": "",
"reviewerId": "110511108367107737234",
"reviewerUrl": "https://www.google.com/maps/contrib/110511108367107737234/reviews?hl=en-US",
"reviewerPhotoUrl": "https://lh3.googleusercontent.com/a-/ALV-UjVDOx8QHSyHNYBimKQAy2b185lLtLhNd-uhwGuh6AbuqUvkPH65=w72-h72-p-rp-mo-ba4-br100",
"reviewerNumberOfReviews": 0,
"isLocalGuide": true,
"reviewOrigin": "Google",
"stars": 5,
"rating": null,
"responseFromOwnerDate": null,
"responseFromOwnerText": null,
"reviewImageUrls": [
"https://lh3.googleusercontent.com/geougc-cs/AIHozJK4iaCdo4hQkehYZn-i8dt1B4ZjvtpD453dVz_gqYNLzdWpJ-ITv2aXJnfXiTm4GgnB7lY_sUfpDWJSONdCiJS6IhwNvmognPjbKF_panMyZK-Kz1CVUUObomtxnxjtRa_s6IKGFA=w600-h450-p",
"https://lh3.googleusercontent.com/geougc-cs/AIHozJIY3WkO9mXfB8L0nLiHAQMmKMGQwXgQsXWGRcWzgcfeGSLLawnuSis--BO35MR49zZYYR-iq4wGjM58zVbSzHBGlmGBHI0wKENVjGIAHRX7rCrqnuYzbnPcfrNuaGAIf8Y6G-2W=w600-h450-p",
"https://lh3.googleusercontent.com/geougc-cs/AIHozJLo94qIiyPmUorcS_YJuJXa1cgQaQwbdIewgZpsqbF-e4e6Vaa1DdOj3wD8Zdh9EeH1tPYCMd4_BTqM0eIgSQ-wPsrNgjvCJ8rPGBFFBaxqo_a9X4qzcx9d1-OiFDz_cDpIrTjmjA=w600-h450-p",
"https://lh3.googleusercontent.com/geougc-cs/AIHozJIFNsk3L0IKaIcL_YsxGUPOZXIzn7iRohFepONb-6nCT72EQ2Tbxx5McohqMxP-xOUV_Pg-PB5k9zfhRT8cSvnZlMhUcRYKOPvUSjaToVhk4jUER-eRr8abU6GjVD4ek_7t8ZA=w600-h450-p"
],
"reviewContext": {},
"reviewDetailedRating": {}
},
{
"name": "Aaron Decker",
"text": "What a wonderful coffee shop! Nice atmosphere, ducks!, delicious breakfast sandwich, and of course amazing coffee to hit the spot! ...",
"textTranslated": null,
"publishAt": "2 months ago",
"publishedAtDate": "2025-01-22T02:18:42.734Z",
"likesCount": 0,
"reviewId": "ChdDSUhNMG9nS0VJQ0FnSUNmZ1oydHpnRRAB",
"reviewUrl": "",
"reviewerId": "100124563986259402322",
"reviewerUrl": "https://www.google.com/maps/contrib/100124563986259402322/reviews?hl=en-US",
"reviewerPhotoUrl": "https://lh3.googleusercontent.com/a-/ALV-UjUWve1upybVvSo9jDd3PgnICSzEzWA_a-2qe42CwF39cRQhSxk0kA=w72-h72-p-rp-mo-ba8-br100",
"reviewerNumberOfReviews": 0,
"isLocalGuide": true,
"reviewOrigin": "Google",
"stars": 5,
"rating": null,
"responseFromOwnerDate": null,
"responseFromOwnerText": null,
"reviewImageUrls": [
"https://lh3.googleusercontent.com/geougc-cs/AIHozJLCeYqMIDXrmqFB9CZusxIHvszUr5x4KM1UXYtENgEk1Nxnq_ibqRDt4k9Rg0eHO1ABxsNyYvjtFlrfCKEKCxlgLeQRb32ilxxsiyuiQ9xrvg4pwXmvwTcFEaG757xOjC--gYEI=w600-h450-p",
"https://lh3.googleusercontent.com/geougc-cs/AIHozJJ3EtJgsuvBz3Ztu0aVwzFAQeoA1Om7ZEqtnWUHjl_ccJXDQglN758tRbZk9lgr2vjWbHTn7Jzl6HNpqCpLY3CeOaPWelhrZT-McegUkJRpg27J1GJDQHZWDVBUBK3sRoR5awHCZw=w600-h450-p",
"https://lh3.googleusercontent.com/geougc-cs/AIHozJI-4MbygMbRsixyGadfJMhB3vVL9lJggmAFrVnUlVjtf_1PgRc3-pq7n-T2rkO4GlhseR23Z221-E4jhq6AWD2NIi9J6jomHwpEIBMJk4Aq8636bULXj_gcgdfd5WX514dyaHDB=w600-h450-p",
"https://lh3.googleusercontent.com/geougc-cs/AIHozJIUc-xaKwVfmIMBmrlg4ITj4lXjM2LsNZJg1STUv-NO8OH--bVoUkSzQQmLwqgbCcHh-sbpabwZezlXHbHgU_x-bF3ZnKWtcSX5RqZNL2LXX6qo3umAdymoQ2eym6-csQxAENhsCg=w600-h450-p"
],
"reviewContext": {},
"reviewDetailedRating": {}
},
{
"name": "Stephanie D",
"text": "Stopped in around 9am on a Saturday. They were a bit busy but they were able to move the line quickly and there is plenty to look at while you wait. Our coffee was delicious but our croissant was even better. We got a bacon, egg and cheese ...",
"textTranslated": null,
"publishAt": "5 months ago",
"publishedAtDate": "2024-10-22T01:18:42.734Z",
"likesCount": 1,
"reviewId": "ChZDSUhNMG9nS0VJQ0FnSUNYbXFpbUJ3EAE",
"reviewUrl": "",
"reviewerId": "100254141532590923077",
"reviewerUrl": "https://www.google.com/maps/contrib/100254141532590923077/reviews?hl=en-US",
"reviewerPhotoUrl": "https://lh3.googleusercontent.com/a-/ALV-UjVC1QNXDVsl2RyOM9UNQelqeF3pFdyTb0RvviZmA7_3n5NTRrt3=w72-h72-p-rp-mo-ba4-br100",
"reviewerNumberOfReviews": 0,
"isLocalGuide": true,
"reviewOrigin": "Google",
"stars": 4,
"rating": null,
"responseFromOwnerDate": null,
"responseFromOwnerText": null,
"reviewImageUrls": [
"https://lh3.googleusercontent.com/geougc-cs/AIHozJJct2N0-nz5ADJcFJ6zoT80k7xaSgq1MHl8AqO31k9-D_a6x90qciLdCni8vUQkwOO4iUIn_Bn5wiG1FecOafyRESitBfnxENj_pGdqRNroR9VvFbLXhRwLqMiXyIogWGZazx0=w600-h900-p",
"https://lh3.googleusercontent.com/geougc-cs/AIHozJJoGd5NaZWjBrDCaov1klIkKsCOFfHuEAMQwgMqzTOBN9SRkMNKSO7b6TW9jxFWeYjJsFkf9GBm2rJy0BTOqibTRBH-CQ1CA8e-Bm9wMrUnjNFiLUngrj40c_4F78RxS1IQugK0xQ=w600-h450-p",
"https://lh3.googleusercontent.com/geougc-cs/AIHozJKWgd4ch6wzczxPuSKedrkH6rgkUgJ5WeJRlscbhV6Lxn76qyLhcjaSPbOMTYikfal7XI5T0H9vRUdKqc1ft0X7LYLg8oV4ElgDNbRWYOpluLfvgdL0yiDqkdfpyRMsUWONzDAq=w600-h450-p"
],
"reviewContext": {},
"reviewDetailedRating": {}
}
],
"userPlaceNote": null,
"restaurantData": {}
}
```
Olostep provides a hosted JSON file with the structured location data. You can access the JSON file using the json\_hosted\_url field in the response:
* Structured JSON: [View example JSON](https://olostep-storage.s3.us-east-1.amazonaws.com/json_y6jwelcpdl.json)
If you want to also get the HTML and Markdown content of the location data, you can include these formats in the formats parameter and Olostep will return them in the response and provide hosted URLs for each format.
## Use Cases
The Google Maps parser is particularly useful for:
Business Intelligence: Collect data about competitors' locations, ratings, and customer feedback
Market Research: Analyze customer reviews and sentiments about businesses in specific areas
Location-Based Applications: Integrate detailed business information and customer reviews
Real Estate Analysis: Gather data about nearby amenities and local businesses
Local SEO: Monitor business presence and customer reviews on Google Maps
## Important Notes
Language Parameter: The ?hl=en parameter sets the language to English. Adjust as needed for other languages.
## Conclusion
Using the Google Maps parser with Olostep's API allows you to extract structured location data that can be easily integrated into your applications or used for analysis. The parser handles the complex task of extracting information from Google Maps pages, providing you with clean, structured data.
To get information about available parsers or to request a custom parser for your specific use case, please contact us at `info@olostep.com`
# Get Google Search Results (JS)
Source: https://docs.olostep.com/examples/google-search-js
SERP API to scrape Google search results using Olostep and a Javascript code snippet.
# Google Search Scraper with Olostep
This guide demonstrates how to use the Olostep API to scrape Google search results and parse them into structured JSON data. This is particularly useful for automating research tasks, gathering competitive intelligence, or building applications that require search data.
## How It Works
The example below in Javascript shows how to search for a LinkedIn profile URL of a specific person (Patrick Collison) using Google search and Olostep's google search parser `@olostep/google-search`
```javascript theme={null}
async function scrapeGoogleSearch(apiKey, query = "site%3Alinkedin.com+Patrick+Collison") {
const endpoint = "https://api.olostep.com/v1/scrapes";
const payload = {
"formats": ["json"],
"parser": {"id": "@olostep/google-search"},
"url_to_scrape": `https://www.google.com/search?q=${encodeURIComponent(query)}&gl=us&hl=en`,
"wait_before_scraping": 0,
};
const headers = {
"Authorization": `Bearer ${apiKey}`,
"Content-Type": "application/json"
};
try {
const response = await fetch(endpoint, {
method: "POST",
headers: headers,
body: JSON.stringify(payload)
});
const data = await response.json();
console.log(JSON.stringify(data, null, 4));
return data;
} catch (error) {
console.error("Error:", error);
throw error;
}
}
// Replace with your actual Olostep API key
scrapeGoogleSearch("");
```
## Response Format
When you make a request to the Olostep API with the Google Search parser, you'll receive a JSON response like the example below:
```json theme={null}
{
"id": "scrape_f2xghz17kt",
"object": "scrape",
"created": 1742679301,
"metadata": {},
"retrieve_id": "f2xghz17kt",
"url_to_scrape": "https://www.google.com/search?q=site%253Alinkedin.com%2BPatrick%2BCollison&gl=us&hl=en",
"result": {
"html_content": null,
"markdown_content": null,
"text_content": null,
"json_content": "{\"searchParameters\":{\"type\":\"search\",\"engine\":\"google\",\"q\":\"site:linkedin.com Patrick Collison\"},\"knowledgeGraph\":{\"description\":\"Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...\"},\"organic\":[{\"title\":\"Patrick Collison - Stripe\",\"link\":\"https://www.linkedin.com/in/patrickcollison\",\"position\":1,\"snippet\":\"Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...\",\"meta\":\"10.8K+ followers\"},{\"title\":\"The Stripe Story: How Patrick Collison Revolutionized ...\",\"link\":\"https://www.linkedin.com/pulse/stripe-story-how-patrick-collison-revolutionized-online-anshuman-jha-jzzic\",\"position\":2,\"snippet\":\"The Early Years: A Genius in the Making. Patrick Collison wasn't just bright—he was a supernova. By age 10, he'd devoured university-level math ...\"},{\"title\":\"In 2005, Patrick Collison was a 16-year-old winning ...\",\"link\":\"https://www.linkedin.com/posts/itselanagold_in-2005-patrick-collison-was-a-16-year-old-activity-7308533537576497154-w5vC\",\"position\":3,\"snippet\":\"In 2005, Patrick Collison was a 16-year-old winning Ireland's Young Scientist of the Year competition. By 2008, he and his younger brother ...\"},{\"title\":\"Patrick Collison on the importance of waiting a really long ...\",\"link\":\"https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-waiting-activity-7286001819145707520-1mdI\",\"position\":4,\"snippet\":\"Patrick argues you should also view every person you hire as bringing along another 50 people just like them if your company is successful.\"},{\"title\":\"Tim Ferriss' Post - Patrick Collison — CEO of Stripe (#353)\",\"link\":\"https://www.linkedin.com/posts/timferriss_patrick-collison-ceo-of-stripe-353-activity-7271892372358148096--dsK\",\"position\":5,\"snippet\":\"Author of 5 #1 NYT/WSJ bestsellers, early-stage investor, host of The Tim Ferriss Show podcast (1B+ downloads), and collector of the strange.\"},{\"title\":\"Patrick Collison wanted a guide to Stripe's culture that ...\",\"link\":\"https://www.linkedin.com/posts/first-round-capital_patrick-collison-wanted-a-guide-to-stripes-activity-7304833456948097024-Tt6h\",\"position\":6,\"snippet\":\"Patrick Collison wanted a guide to Stripe's culture that convinced 50% of candidates not to join. And Eeke de Milliano was tasked with ...\"},{\"title\":\"The Collison brothers (John & Patrick) explain why Stripe is ...\",\"link\":\"https://www.linkedin.com/posts/marcelvanoost_the-collison-brothers-john-patrick-explain-activity-7301586346349850624-L-4U\",\"position\":7,\"snippet\":\"The Collison brothers (John & Patrick) explain why Stripe is better off staying Private: \\\" This is our life's work. We're not going anywhere ...\"},{\"title\":\"Stripe CEO Patrick Collison on Crafting a Culture ...\",\"link\":\"https://www.linkedin.com/posts/jennifer-chatman-8086a918_stripe-ceo-patrick-collison-on-crafting-a-activity-7231753022849085440-0RE5\",\"position\":8,\"snippet\":\"When Patrick Collison and his brother John Collison founded digital payment company Stripe in 2010, he didn't come in with “any kind of ...\"},{\"title\":\"Patrick Collison on the importance of beauty and ...\",\"link\":\"https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-beauty-activity-7247935993817751552-Qt6h\",\"position\":9,\"snippet\":\"Patrick Collison on the importance of beauty and craftsmanship when building products “If Stripe is a monstrously successful business, ...\"}]}",
"llm_extract": null,
"screenshot_hosted_url": null,
"html_hosted_url": null,
"markdown_hosted_url": null,
"json_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json_f2xghz17kt.json",
"text_hosted_url": null,
"links_on_page": [],
"page_metadata": {
"status_code": 200,
"title": ""
}
}
}
```
The response contains:
* **Basic request information**: `id`, `object`, `created` timestamp, `url_to_scrape`
* **Result object** with URLs to access different formats of the data
* **json\_content** with structured search results including:
* `searchParameters`: Information about the search query
* `knowledgeGraph`: Detailed information about the search subject (when available)
* `organic`: List of search results with title, link, position, and snippet
* `peopleAlsoAsk`: Related questions that users commonly search for
* `relatedSearches`: Suggested related search queries
`json_content` is the main part of the response with the structured search results. You can access the JSON content directly from the response or use the hosted URL provided in the response.
## Structured Response: json\_content
```json theme={null}
{
"searchParameters": {
"type": "search",
"engine": "google",
"q": "site:linkedin.com Patrick Collison"
},
"knowledgeGraph": {
"description": "Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ..."
},
"organic": [
{
"title": "Patrick Collison - Stripe",
"link": "https://www.linkedin.com/in/patrickcollison",
"position": 1,
"snippet": "Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...",
"meta": "10.8K+ followers"
},
{
"title": "The Stripe Story: How Patrick Collison Revolutionized ...",
"link": "https://www.linkedin.com/pulse/stripe-story-how-patrick-collison-revolutionized-online-anshuman-jha-jzzic",
"position": 2,
"snippet": "The Early Years: A Genius in the Making. Patrick Collison wasn't just bright—he was a supernova. By age 10, he'd devoured university-level math ..."
},
{
"title": "In 2005, Patrick Collison was a 16-year-old winning ...",
"link": "https://www.linkedin.com/posts/itselanagold_in-2005-patrick-collison-was-a-16-year-old-activity-7308533537576497154-w5vC",
"position": 3,
"snippet": "In 2005, Patrick Collison was a 16-year-old winning Ireland's Young Scientist of the Year competition. By 2008, he and his younger brother ..."
},
{
"title": "The Collison brothers (John & Patrick) explain why Stripe is ...",
"link": "https://www.linkedin.com/posts/marcelvanoost_the-collison-brothers-john-patrick-explain-activity-7301586346349850624-L-4U",
"position": 4,
"snippet": "The Collison brothers (John & Patrick) explain why Stripe is better off staying Private: \" This is our life's work. We're not going anywhere ..."
},
{
"title": "Patrick Collison on the importance of waiting a really long ...",
"link": "https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-waiting-activity-7286001819145707520-1mdI",
"position": 5,
"snippet": "Patrick argues you should also view every person you hire as bringing along another 50 people just like them if your company is successful."
},
{
"title": "Tim Ferriss' Post - Patrick Collison — CEO of Stripe (#353)",
"link": "https://www.linkedin.com/posts/timferriss_patrick-collison-ceo-of-stripe-353-activity-7271892372358148096--dsK",
"position": 6,
"snippet": "Author of 5 #1 NYT/WSJ bestsellers, early-stage investor, host of The Tim Ferriss Show podcast (1B+ downloads), and collector of the strange."
},
{
"title": "Patrick Collison on the importance of beauty and ...",
"link": "https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-beauty-activity-7247935993817751552-Qt6h",
"position": 7,
"snippet": "Patrick Collison on the importance of beauty and craftsmanship when building products "If Stripe is a monstrously successful business, ..."
},
{
"title": "Stripe founder Patrick Collison tells the story of almost ...",
"link": "https://www.linkedin.com/posts/the-startup-archive_stripe-founder-patrick-collison-tells-the-activity-7235977194211000321-V-Cd",
"position": 8,
"snippet": "Stripe founder Patrick Collison tells the story of almost naming the company PayDemon Patrick and John Collison founded Stripe in 2010 when ..."
},
{
"title": "Patrick Collison created $50 billion of value as a co- ...",
"link": "https://www.linkedin.com/posts/tom-alder_patrick-collison-created-50-billion-of-value-activity-7239241304780513281-isxK",
"position": 9,
"snippet": "Patrick Collison created $50 billion of value as a co-founder of Stripe. He has also built the largest carbon removal program in the world."
}
]
}
```
Olostep provides also a hosted JSON file with the structured search results. You can access the JSON file using the `json_hosted_url` field in the response:
* Structured JSON: [View example JSON](https://olostep-storage.s3.us-east-1.amazonaws.com/json_f2xghz17kt.json)
If you want to also get the HTML and Markdown content of the search results, you can include these formats in the `formats` parameter and Olostep will return them in the response and provide hosted URLs for each format.
* [Markdown format](https://olostep-storage.s3.us-east-1.amazonaws.com/markDown_f2xghz17kt.txt)
* [HTML](https://olostep-storage.s3.us-east-1.amazonaws.com/text_f2xghz17kt.txt)
## Example Usage Scenarios
### 1. Finding LinkedIn Profiles
In the example above, we're searching for Patrick Collison's LinkedIn profile by using the search query `site:linkedin.com Patrick Collison`. This technique can be used to find professional profiles for any person.
### 2. Researching Companies
You can modify the query to search for company information:
```javascript theme={null}
// Search for company information
scrapeGoogleSearch(apiKey, "Stripe company information revenue");
```
### 3. Tracking News Articles
Find recent news about a specific topic:
```javascript theme={null}
// Search for recent news about AI
scrapeGoogleSearch(apiKey, "artificial intelligence news after:2023-01-01");
```
### 4. Competitive Analysis
Monitor competitors' online presence:
```javascript theme={null}
// Search for competitor mentions
scrapeGoogleSearch(apiKey, "\"Company X\" vs \"Company Y\" comparison");
```
## Important Notes
4. **Search Parameters**: The `gl=us` and `hl=en` parameters set the geolocation to US and language to English. Adjust these as needed.
## Conclusion
Once you have the search results data, you can:
1. Parse specific data points of interest
2. Store the results in a database
3. Build a custom search interface
4. Trigger actions based on search findings
5. Integrate with other APIs or services
If you need to extract different data points or customize the search behavior, please get in touch at '[info@olostep.com](mailto:info@olostep.com)\`
# Price Tracking with Olostep
Source: https://docs.olostep.com/examples/price-monitoring
Learn how to use Olostep to track product prices at scale on an e-commerce.
## Overview
Olostep provides a web scraping API that enables real-time price tracking of millions of products on an e-commerce at regular intervals (e.g. every few hours) in a scalable and cost effective way.
This is useful for businesses that want to monitor price fluctuations, compare prices across multiple websites, or track competitor pricing strategies.
In this guide, we will see how a customer is using Olostep to set up automated price tracking for millions of Amazon products daily.
## Why Use Olostep for Price Tracking?
* **Scalability:** Track prices for millions of products every few hours.
* **Automation:** Set up scheduled scraping tasks that run at predefined times/regular intervals.
* **Multiple Formats:** Retrieve data in JSON, html or markdown format.
* **Custom Parsers:** Extract only the relevant JSON information with our parsers or pass your own to the API.
## How to Track Prices Using Olostep
### Overview of the Process Setup
When tracking products at scale we recommend using Olostep's [Batches endpoint](https://docs.olostep.com/api-reference/batches/create).
This endpoint allows you to send multiple batches of URLs (each of up to 10k) to be processed in parallel and then retrieve the results after 5-8 minutes. You can send multiple batches at the same time, monitor their progress and retrieve the results once they are complete. In this way you can process millions of URLs in 15-20 minutes.
The overall flow for price tracking using Olostep is as follows:
1. **Read the products from the database and save the URLs you want to track in a CSV file.**
2. **Read the data from the CSV file and start a batch using Olostep's batch endpoint.** This is done by posting the data to the endpoint in chunks of up to 10,000 URLs at a time.
3. **Check the batch status every 60 seconds** to monitor the progress.
4. **Once the batch is complete, read the content and use it in your workflow.**
You can start a batch and be returned the html/markdown content of the page and then parse it yourself to extract the data you want. But we recommend starting the batch with a parser so you are returned a JSON object with only the parsed data you need.
You can pass your own parser to the API or use one of the pre-built ones we have for some common websites (e.g Amazon product pages, Google search results, Linkedin profiles, etc...).
We store the data for each batch for 7 days so you can retrieve it multiple times if needed.
### Step 1: Export Product Data from your Database
The first step is to retrieve product information from your database and save it in a CSV format. This file should contain product identifiers, URLs, and any additional metadata required for tracking.
### Step 2: Start a Batch with Olostep
To start a batch, read the product data from the CSV and send it to the Olostep batch endpoint. This is done using an HTTP POST request with a JSON payload.
Each batch can have up to 10k URLs. For large datasets (>10,000 URLs), split into multiple batches and send them in parallel.
A batch consists of an array of items, where each item represents a product URL to be processed. Here's the structure of a batch request
```python theme={null}
import requests
def start_batch(batch_array):
payload = {
"batch_array": batch_array, # Array of items to process
"batch_country": "IT", # Country code for the batch
"parser": "@olostep/amazon-it-product" # Optional: Specify a custom parser so you only get the JSON data you need
}
headers = {
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json"
}
response = requests.post(
"https://api.olostep.com/v1/batches",
headers=headers,
json=payload
)
return response.json()["id"]
```
#### Batch Array Structure
Each item in the batch\_array should follow this structure:
```json filename="batch_item.json" theme={null}
{
"custom_id": "unique_identifier", // Required: Your unique identifier for the item
"url": "product_url", // Required: URL to be processed
"wait_before_scraping": 0, // Optional: Wait time before scraping each url in the batch (in milliseconds)
}
```
Parameters
Array of items to process. Maximum of 10,000 URLs per batch. Each item must have a unique `custom_id`.
Two-letter country code (e.g., "IT" for Italy).
Name of the custom parser to use (e.g., "@olostep/amazon-it-product"). Contact us at [info@olostep.com](mailto:info@olostep.com) to get access to the pre-built parsers or to create your own.
Response
```json filename="response.json" theme={null}
{
"id": "batch_54ikwskmt8"
}
```
The endpoint returns a JSON object containing the batch\_id, which can be used to monitor the status and then retrieve the results.
Example Usage
```python theme={null}
# Prepare batch array
batch_array = [
{
"custom_id": "product_123",
"url": "https://www.amazon.it/dp/B0CHF6Z393/?coliid=INQXTGFQF4FM4&colid=1R0NGA5NR5LSZ&psc=1&ref_=list_c_wl_lv_vv_lig_dp_it"
},
{
"custom_id": "product_124",
"url": "https://www.amazon.it/dp/B0CHMJL774/?coliid=I6CFYA5EHVHE2&colid=1R0NGA5NR5LSZ&psc=1&ref_=list_c_wl_lv_vv_lig_dp_it"
}
]
# Start batch processing
batch_id = start_batch(batch_array)
print(f"Started batch: {batch_id}")
```
### Step 3: Monitor Batch Status
Once a batch is started, you'll need to monitor its status to determine when processing is complete. The API provides a status endpoint that can be polled periodically (e.g., every 60 seconds) with the batch\_id
```python filename="check_status.py" theme={null}
import requests
def check_batch_status(batch_id):
headers = {"Authorization": "Bearer " + API_KEY}
response = requests.request(
"GET",
f"https://api.olostep.com/v1/batches/{batch_id}",
headers=headers
)
return response.json()["status"]
```
For production use, it's recommended to implement asynchronous monitoring to handle multiple batches efficiently:
```python theme={null}
import asyncio
async def monitor_batch(batch_id: str) -> None:
"""Monitor a single batch until it's completed"""
while True:
status = check_batch_status(batch_id)
if status == "completed":
print(f"Batch {batch_id} completed!")
return
print(f"Batch {batch_id} still processing... Checking again in 60 seconds")
await asyncio.sleep(60)
```
### Step 4: Retrieve the IDs for Completed Items
Once the batch is marked as completed, you can fetch the list of completed items. Each item will have a retrieve\_id. If you want the actual content use the retrieve endpoint by passing the `retrieve_id`
```python theme={null}
import requests
def get_completed_items(batch_id):
headers = {"Authorization": "Bearer YOUR_API_KEY"}
response = requests.get(f"https://api.olostep.com/v1/batches/{batch_id}/items", headers=headers)
return response.json()["items"]
```
This will return the completed items that have each a `retrieve_id` for every URL sent. You can then use the retrieve endpoint to retrieve and store the extracted data (html, markdown or JSON) for each URL.
You can get the `retrieve_id` for each item in the batch using the following code:
```python theme={null}
items = get_completed_items("your_batch_id")
for item in items:
print(f"""
URL: {item['url']}
Custom ID: {item['custom_id']}
Retrieve ID: {item['retrieve_id']}
---
""")
```
### Step 5: Retrieve the Content for each Item
Once you have the `retrieve_id` for each item, you can fetch its content (HTML, Markdown, or JSON) using the retrieve endpoint:
```python filename="retrieve_content.py" theme={null}
def retrieve_content(retrieve_id):
url = "https://api.olostep.com/v1/retrieve"
headers = {"Authorization": "Bearer YOUR_API_KEY"}
params = {"retrieve_id": retrieve_id}
response = requests.get(
url,
headers=headers,
params=params
)
return response.json()
# Example usage
retrieve_id = "product_123"
content = retrieve_content(retrieve_id)
# If you want to process multiple items
def process_batch_content(batch_id):
items = get_completed_items(batch_id)
for item in items:
content = retrieve_content(item['retrieve_id'])
# Process or store the content as needed
```
## Conclusion
By following these steps, you can set up an automated price-tracking system using Olostep. Shortly we will publish an open-source github repo with the full code for this example.
# Get Google Search Results (python)
Source: https://docs.olostep.com/examples/serp
SERP API to scrape Google search results using Olostep and a Python code snippet.
## Overview
Olostep's API allows you to extract structured data from search engine results by using [parsers](../../features/structured-content/parsers). These parsers transform the raw HTML of search results into clean, structured JSON data that's ready for analysis or integration into your applications.
## Integration Example
To get parsed JSON content from search results, you need to include `json` in the `formats` parameter and specify the name of the parser `@olostep/google-search` in the `parser` object.
Here's how to retrieve Google search results in a structured format:
```python theme={null}
import requests
import json
endpoint = "https://api.olostep.com/v1/scrapes"
payload = {
"formats": ["json"],
"parser": {"id": "@olostep/google-search"},
"url_to_scrape": "https://www.google.com/search?q=alexander+the+great&gl=us&hl=en",
"wait_before_scraping": 0,
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
response = requests.request("POST", endpoint, json=payload, headers=headers)
print(json.dumps(response.json(), indent=4))
```
## Response Format
When you make a request to the Olostep API with the Google Search parser, you'll receive a JSON response like the example below:
```json theme={null}
{
"id": "scrape_94iqy385ty",
"object": "scrape",
"created": 1740595134,
"metadata": {},
"retrieve_id": "94iqy385ty",
"url_to_scrape": "https://www.google.com/search?q=alexander+the+great&gl=us&hl=en",
"result": {
"html_content": null,
"markdown_content": null,
"text_content": null,
"json_content": "{\"searchParameters\":{\"type\":\"search\",\"engine\":\"google\",\"q\":\"alexander the great\"},\"knowledgeGraph\":{\"title\":\"Alexander the Great\",\"type\":\"Former King of Macedonia\",\"description\":\"Alexander III of Macedon, most commonly known as Alexander the Great, was a king of the ancient Greek kingdom of Macedon.\",\"imageUrl\":\"https://www.mayaincaaztec.com/ancient-greece/alexander-the-great\",\"attributes\":{\"Born\":\"July 356 BC, Pella\",\"Died\":\"June 323 BC (age 32 years), Babylon\",\"Spouse\":\"Roxana (m. 327 BC\u2013323 BC), Parysatis II (m. 324 BC\u2013323 BC), Stateira (m. 324 BC\u2013323 BC)\",\"Children\":\"Alexander IV of Macedon\",\"Full name\":\"Alexander III of Macedon\",\"Siblings\":\"Cleopatra of Macedon, Philip III of Macedon, Thessalonike of Macedon, Cynane, Caranus, Europa of Macedon\"}},\"organic\":[{\"title\":\"Alexander the Great\",\"link\":\"https://en.wikipedia.org/wiki/Alexander_the_Great\",\"position\":1,\"snippet\":\"He was undefeated in battle and is widely considered to be one of history's greatest and most successful military commanders.\",\"sitelinks\":[{\"title\":\"Death of Alexander the Great\",\"link\":\"https://en.wikipedia.org/wiki/Death_of_Alexander_the_Great\"},{\"title\":\"Wars of Alexander the Great\",\"link\":\"https://en.wikipedia.org/wiki/Wars_of_Alexander_the_Great\"}]},{\"title\":\"Alexander the Great | Biography, Empire, Death, & Facts\",\"link\":\"https://www.britannica.com/biography/Alexander-the-Great#:~:text=Top%20Questions-,Why%20is%20Alexander%20the%20Great%20famous%3F,Greece%20to%20part%20of%20India.\",\"position\":2},{\"title\":\"Alexander the Great's Last Three Wishes. - LinkedIn\",\"link\":\"https://www.linkedin.com/pulse/moment-can-last-lifetime-alexander-greats-three-wishes-holt#:~:text=1)%20The%20king%20of%20Macedon,my%20coffin%2C%22%20Alexander%20said.\",\"position\":3},{\"title\":\"Alexander the Great Failure: The Collapse of the Macedonian Empire\",\"link\":\"https://www.publishersweekly.com/9781847251886#:~:text=His%20inability%20to%20delegate%20work,its%20independence%20and%20its%20boundaries.\",\"position\":4},{\"title\":\"Who defeated Alexander The Great? Who conquered Greece after him ...\",\"link\":\"https://www.quora.com/Who-defeated-Alexander-The-Great-Who-conquered-Greece-after-him-and-why-were-they-able-to-conquer-that-region-while-Alexander-couldnt#:~:text=No%20one%20defeated%20Alexander%20the,his%20death%20was%20not%20natural.\",\"position\":5},{\"title\":\"Alexander the Great | Biography, Empire, Death, & Facts\",\"link\":\"https://www.britannica.com/biography/Alexander-the-Great\",\"position\":6,\"snippet\":\"Feb 11, 2025 \u2014 Alexander the Great, a fearless Macedonian king and military genius, conquered vast territories from Greece to Egypt and India, ...\"},{\"title\":\"Alexander the Great: Empire & Death\",\"link\":\"https://www.history.com/topics/ancient-greece/alexander-the-great\",\"position\":7,\"snippet\":\"Nov 9, 2009 \u2014 Alexander the Great was an ancient Macedonian ruler and one of history's greatest military minds who, as King of Macedonia and Persia, ...\"},{\"title\":\"History - Alexander the Great\",\"link\":\"https://www.bbc.co.uk/history/historic_figures/alexander_the_great.shtml\",\"position\":8,\"snippet\":\"Alexander III of Macedon, better known as Alexander the Great, single-handedly changed the nature of the ancient world in little more than a decade.\"},{\"title\":\"Alexander the Great - National Geographic Education\",\"link\":\"https://education.nationalgeographic.org/resource/alexander-great/\",\"position\":9,\"snippet\":\"Oct 19, 2023 \u2014 Alexander was born in 356 B.C.E. in Pella, Macedonia, to King Philip II. As a young boy, Alexander was taught to read, write, and play the lyre.\"},{\"title\":\"Who loved Alexander the Great?\",\"link\":\"https://museums.cam.ac.uk/magic/who-loved-alexander-great\",\"position\":10,\"snippet\":\"Throughout his life, Alexander married 3 women and fathered at least 2 children but also had several male lovers. Amongst his closest relationships was that ...\"},{\"title\":\"Alexander the Great (1956)\",\"link\":\"https://www.imdb.com/title/tt0048937/\",\"position\":11,\"snippet\":\"The life and military conquests of Alexander III of Macedon (July 20/21, 356 - June 10/11, 323 B.C.), commonly known as Alexander the Great.\"},{\"title\":\"Alexander the Great\",\"link\":\"https://www.worldhistory.org/Alexander_the_Great/\",\"position\":12,\"snippet\":\"Nov 14, 2013 \u2014 He is known as 'the great' both for his military genius and his diplomatic skills in handling the various populaces of the regions he conquered.\"}],\"peopleAlsoAsk\":[{\"question\":\"What is Alexander the Great most famous for?\"},{\"question\":\"What did Alexander the Great say before he died?\",\"link\":\"https://www.linkedin.com/pulse/moment-can-last-lifetime-alexander-greats-three-wishes-holt#:~:text=1)%20The%20king%20of%20Macedon,my%20coffin%2C%22%20Alexander%20said.\",\"title\":\"Alexander the Great's Last Three Wishes. - LinkedIn\"},{\"question\":\"What led to the fall of Alexander?\",\"link\":\"https://www.publishersweekly.com/9781847251886#:~:text=His%20inability%20to%20delegate%20work,its%20independence%20and%20its%20boundaries.\",\"title\":\"Alexander the Great Failure: The Collapse of the Macedonian Empire\"},{\"question\":\"Which country defeated Alexander the Great?\",\"link\":\"https://www.quora.com/Who-defeated-Alexander-The-Great-Who-conquered-Greece-after-him-and-why-were-they-able-to-conquer-that-region-while-Alexander-couldnt#:~:text=No%20one%20defeated%20Alexander%20the,his%20death%20was%20not%20natural.\",\"title\":\"Who defeated Alexander The Great? Who conquered Greece after him ...\"}],\"relatedSearches\":[{\"query\":\"Alexander the Great book\"},{\"query\":\"Alexander the Great empire\"},{\"query\":\"Alexander the Great death\"},{\"query\":\"Alexander the Great religion\"},{\"query\":\"Alexander the Great Empire map\"},{\"query\":\"Alexander the Great achievements\"},{\"query\":\"What was Alexander the Great known for\"},{\"query\":\"Alexander the Great empire name\"}]}",
"llm_extract": null,
"screenshot_hosted_url": null,
"html_hosted_url": null,
"markdown_hosted_url": null,
"json_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json_94iqy385ty.json",
"text_hosted_url": null,
"links_on_page": [],
"page_metadata": {
"status_code": 200,
"title": ""
}
}
}
```
The response contains:
* **Basic request information**: `id`, `object`, `created` timestamp, `url_to_scrape`
* **Result object** with URLs to access different formats of the data
* **json\_content** with structured search results including:
* `searchParameters`: Information about the search query
* `knowledgeGraph`: Detailed information about the search subject (when available)
* `organic`: List of search results with title, link, position, and snippet
* `peopleAlsoAsk`: Related questions that users commonly search for
* `relatedSearches`: Suggested related search queries
`json_content` is the main part of the response with the structured search results. You can access the JSON content directly from the response or use the hosted URL provided in the response.
## Structured Response: json\_content
```json theme={null}
{
"searchParameters": {
"type": "search",
"engine": "google",
"q": "alexander the great"
},
"knowledgeGraph": {
"title": "Alexander the Great",
"type": "Former King of Macedonia",
"description": "Alexander III of Macedon, most commonly known as Alexander the Great, was a king of the ancient Greek kingdom of Macedon.",
"imageUrl": "https://www.mayaincaaztec.com/ancient-greece/alexander-the-great",
"attributes": {
"Born": "July 356 BC, Pella",
"Died": "June 323 BC (age 32 years), Babylon",
"Spouse": "Roxana (m. 327 BC–323 BC), Parysatis II (m. 324 BC–323 BC), Stateira (m. 324 BC–323 BC)",
"Children": "Alexander IV of Macedon",
"Full name": "Alexander III of Macedon",
"Siblings": "Cleopatra of Macedon, Philip III of Macedon, Thessalonike of Macedon, Cynane, Caranus, Europa of Macedon"
}
},
"organic": [
{
"title": "Alexander the Great",
"link": "https://en.wikipedia.org/wiki/Alexander_the_Great",
"position": 1,
"snippet": "He was undefeated in battle and is widely considered to be one of history's greatest and most successful military commanders.",
"sitelinks": [
{
"title": "Death of Alexander the Great",
"link": "https://en.wikipedia.org/wiki/Death_of_Alexander_the_Great"
},
{
"title": "Wars of Alexander the Great",
"link": "https://en.wikipedia.org/wiki/Wars_of_Alexander_the_Great"
}
]
},
{
"title": "Alexander the Great Failure: The Collapse of the Macedonian Empire",
"link": "https://www.publishersweekly.com/9781847251886#:~:text=His%20inability%20to%20delegate%20work,its%20independence%20and%20its%20boundaries.",
"position": 2
},
{
"title": "Which Indian king first time defeated Alexander? - Quora",
"link": "https://www.quora.com/Which-Indian-king-first-time-defeated-Alexander#:~:text=Alexander%20the%20Great%20was%20defeated,returned%20with%20only%2020000%20troops.",
"position": 3
},
{
"title": "Alexander the Great | Biography, Empire, Death, & Facts",
"link": "https://www.britannica.com/biography/Alexander-the-Great",
"position": 4,
"snippet": "Feb 11, 2025 — Alexander the Great, a fearless Macedonian king and military genius, conquered vast territories from Greece to Egypt and India, ..."
},
{
"title": "Alexander the Great: Empire & Death",
"link": "https://www.history.com/topics/ancient-greece/alexander-the-great",
"position": 5,
"snippet": "Nov 9, 2009 — Alexander the Great was an ancient Macedonian ruler and one of history's greatest military minds who, as King of Macedonia and Persia, ..."
},
{
"title": "Alexander the Great (1956)",
"link": "https://www.imdb.com/title/tt0048937/",
"position": 6,
"snippet": "The life and military conquests of Alexander III of Macedon (July 20/21, 356 - June 10/11, 323 B.C.), commonly known as Alexander the Great."
},
{
"title": "History - Alexander the Great",
"link": "https://www.bbc.co.uk/history/historic_figures/alexander_the_great.shtml",
"position": 7,
"snippet": "Alexander III of Macedon, better known as Alexander the Great, single-handedly changed the nature of the ancient world in little more than a decade."
},
{
"title": "Who loved Alexander the Great?",
"link": "https://museums.cam.ac.uk/magic/who-loved-alexander-great",
"position": 8,
"snippet": "Throughout his life, Alexander married 3 women and fathered at least 2 children but also had several male lovers. Amongst his closest relationships was that ..."
},
{
"title": "Alexander the Great",
"link": "https://www.worldhistory.org/Alexander_the_Great/",
"position": 9,
"snippet": "Nov 14, 2013 — He is known as 'the great' both for his military genius and his diplomatic skills in handling the various populaces of the regions he conquered."
},
{
"title": "Alexander the Great - National Geographic Education",
"link": "https://education.nationalgeographic.org/resource/alexander-great/",
"position": 10,
"snippet": "Oct 19, 2023 — Alexander was born in 356 B.C.E. in Pella, Macedonia, to King Philip II. As a young boy, Alexander was taught to read, write, and play the lyre."
}
],
"peopleAlsoAsk": [
{
"question": "What is Alexander the Great most famous for?"
},
{
"question": "What did Alexander the Great say before he died?"
},
{
"question": "What led to the fall of Alexander?",
"link": "https://www.publishersweekly.com/9781847251886#:~:text=His%20inability%20to%20delegate%20work,its%20independence%20and%20its%20boundaries.",
"title": "Alexander the Great Failure: The Collapse of the Macedonian Empire"
},
{
"question": "Who first defeated Alexander the Great?",
"link": "https://www.quora.com/Which-Indian-king-first-time-defeated-Alexander#:~:text=Alexander%20the%20Great%20was%20defeated,returned%20with%20only%2020000%20troops.",
"title": "Which Indian king first time defeated Alexander? - Quora"
}
],
"relatedSearches": [
{
"query": "Alexander the Great book"
},
{
"query": "Alexander the Great empire"
},
{
"query": "Alexander the Great death"
},
{
"query": "Alexander the Great religion"
},
{
"query": "Alexander the Great Empire map"
},
{
"query": "Alexander the Great achievements"
},
{
"query": "What was Alexander the Great known for"
},
{
"query": "Alexander the Great movie"
}
]
}
```
Olostep provides also a hosted JSON file with the structured search results. You can access the JSON file using the `json_hosted_url` field in the response:
* Structured JSON: [View example JSON](https://olostep-storage.s3.us-east-1.amazonaws.com/json_vxc86vq2pf.json)
If you want to also get the HTML and Markdown content of the search results, you can include these formats in the `formats` parameter and Olostep will return them in the response and provide hosted URLs for each format.
* [Markdown format](https://olostep-storage.s3.us-east-1.amazonaws.com/markDown_vxc86vq2pf.txt)
* [HTML](https://olostep-storage.s3.us-east-1.amazonaws.com/text_vxc86vq2pf.txt)
## Important Notes
**Search Parameters**: The `gl=us` and `hl=en` parameters set the geolocation to US and language to English. Adjust these as needed.
## Conclusion
Passing custom parsers to the API allows you to only get the structured data you want. This makes it easy to integrate search results into your applications, perform data analysis, or build search-related features.
To get information about available parsers or to request a custom parser for your specific use case, please contact us at [info@olostep.com](mailto:info@olostep.com).
# Agents API
Source: https://docs.olostep.com/features/agents
Build no-code research agents that search, scrape, and crawl the web with Olostep.
Through the Olostep Agents API you can create autonomous research agents that can automate data pipelines and search tasks on a schedule and deliver structured results.
* Web research across sites with automated extraction
* Multi-step workflows with scheduled execution and notifications
* Output to JSON/CSV/Sheets/DB
For availability and details, contact us at [info@olostep.com](mailto:info@olostep.com) or [Contact Sales](https://www.olostep.com/contact-sales).
## Installation
```python Python theme={null}
# pip install requests
import requests
```
```js Node theme={null}
// npm install node-fetch
// ESM
import fetch from 'node-fetch'
// CommonJS
const fetch = require('node-fetch')
```
```bash cURL theme={null}
# macOS: builtin curl is fine
```
## Create an agent
Create an agent with a natural language prompt and a target model.
```python Python theme={null}
API_URL = 'https://api.olostep.com/v1/agents' # endpoint available to select customers
API_KEY = ''
headers = {
'Authorization': f'Bearer {API_KEY}',
'Content-Type': 'application/json'
}
data = {
"prompt": """
Find portfolio companies from https://www.vcsheet.com/funds and
write to a Google Sheet with columns (Fund Name, Fund Website URL,
Fund LinkedIn URL, Portfolio Company Name, Portfolio Company URL,
Portfolio Company LinkedIn URL). Run weekly on Monday at 9:00 AM
and email steve@example.com when new companies are added.
""",
"model": "gpt-4.1"
}
response = requests.post(API_URL, headers=headers, json=data)
result = response.json()
print(result)
```
```js Node theme={null}
const res = await fetch('https://api.olostep.com/v1/agents', {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
prompt: 'Find portfolio companies from https://www.vcsheet.com/funds and write to a Google Sheet...',
model: 'gpt-4.1'
})
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/agents" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Find portfolio companies from https://www.vcsheet.com/funds and write to a Google Sheet...",
"model": "gpt-4.1"
}'
```
## Capabilities
The Olostep agent can:
* **Web Research**: Automatically search and extract data from websites
* **Data Organization**: Structure information into spreadsheets, databases, or other formats
* **Scheduled Execution**: Run tasks on a recurring schedule (daily, weekly, at a predefined time)
* **Multi-step Workflows**: Perform complex, multi-part research tasks autonomously
* **Notifications**: Send email alerts when new data is found or tasks complete
* **Custom Output**: Return data as CSV, JSON, Google Sheet, or directly in your database
## Access
The Agents API is currently available to select customers. To get access:
* Email us at [info@olostep.com](mailto:info@olostep.com)
* Or [contact our sales team](https://www.olostep.com/contact-sales)
## Pricing
Agents pricing is variable. It's billed by outcome. The price is negotiated between the client and the agent. Reach out to [info@olostep.com](mailto:info@olostep.com) for more details
# Answers API
Source: https://docs.olostep.com/features/answers
AI answers backed by live web search, scraping, and crawling — not a stale index.
Through the Olostep `/v1/answers` endpoint you can search the web with natural language and return AI‑powered answers and data in the JSON shape you want. This lets you ground your products on real-world data and sources, enrich data points or spreadsheets
* Ask a question or give AI a data point you want to enrich
* Optional: specify the JSON structure you want back
It will:
* Search, clean, validate and return the data it found on the Web
* Return sources used to generate the answer
* Handle uncertainty with `NOT_FOUND` values when data cannot be verified
For API details, see the [Answers Endpoint API Reference](/api-reference/answers/create).
By default we use a generic web index and a cost‑efficient LLM validator.
Enterprise customers have access to proprietary industry specific web indexes, exclusive private data (including phone numbers and emails) and custom LLM models most suited for their use case. Contact us for access: [info@olostep.com](mailto:info@olostep.com)
## Use cases
The answers endpoint can be used to:
* Ground AI applications on real world data and facts
* Enrich spreadsheets and data points for Recruting, Finance, Consulting, and Sales
Here's a demo of an AI powered spreadsheet powered by the Answers endpoint:
[https://www.olostep.com/demos/spreadsheet-enrich](https://www.olostep.com/demos/spreadsheet-enrich)
## Installation
```python Python theme={null}
pip install olostep
```
```javascript Node theme={null}
npm install olostep
```
```bash cURL theme={null}
# curl is available by default on macOS, Linux, and Windows
```
```javascript Node (API) theme={null}
npm install node-fetch
```
```bash Python (API) theme={null}
pip install requests
```
## Usage
Ask a question and pass a JSON schema to guide the output. You can also not pass the `json` parameter and the API will return a json object with the answer text inside the `result` field.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
answer = client.answers.create(
task="What is the latest book by J.K. Rowling?",
json_format={"book_title": "", "author": "", "release_date": ""},
)
print(answer.json_content)
print(answer.sources)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const answer = await client.answers.create({
task: 'What is the latest book by J.K. Rowling?',
jsonFormat: { book_title: '', author: '', release_date: '' },
})
console.log(answer.json_content)
console.log(answer.sources)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/answers" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"task": "What is the latest book by J.K. Rowling?",
"json": {"book_title": "", "author": "", "release_date": ""}
}'
```
```bash CLI theme={null}
olostep answer "What is the latest book by J.K. Rowling?" \
--json-format '{"book_title":"","author":"","release_date":""}'
```
```js Node (API) theme={null}
const res = await fetch('https://api.olostep.com/v1/answers', {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
task: 'What is the latest book by J.K. Rowling?',
json: { book_title: '', author: '', release_date: '' }
})
})
console.log(await res.json())
```
```python Python (API) theme={null}
import requests, json
endpoint = "https://api.olostep.com/v1/answers"
payload = {
"task": "What is the latest book by J.K. Rowling?",
"json": {"book_title": "", "author": "", "release_date": ""}
}
headers = {"Authorization": "Bearer ", "Content-Type": "application/json"}
response = requests.post(endpoint, json=payload, headers=headers)
print(json.dumps(response.json(), indent=2))
```
## Response
Like other Olostep endpoints, you will receive a `answer` object in response. The `answer` object has a few properties like `id` and `result`.
```json theme={null}
{
"id": "answer_9bi0sbj9xa",
"object": "answer",
"created": 1760327323,
"metadata": {},
"task": "What is the latest book by J.K. Rowling?",
"result": {
"json_content": "{\"book_title\":\"The Hallmarked Man\",\"author\":\"J.K. Rowling (as Robert Galbraith)\",\"release_date\":\"2 September 2025\"}",
"json_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/answer_9bi0sbj9xa.json",
"sources": [
"https://strikefans.com/the-books/",
"https://www.facebook.com/groups/496943608606523/posts/1136830134617864/",
"https://robert-galbraith.com/strike-books/",
"https://www.novelsuspects.com/series-list/robert-galbraith-cormoran-strike-series-in-order/",
"https://www.reddit.com/r/books/comments/1na833a/jk_rowlings_new_strike_novel_900_pages_of_romance/",
"https://www.harrypotter.com/writing-by-jk-rowling",
"https://stories.jkrowling.com/book-news/",
"https://deadline.com/2024/09/jk-rowling-writing-futuristic-novel-1236093909/",
"https://www.reddit.com/r/FantasticBeasts/comments/1cl1shn/jk_rowling_may_2024_ive_got_six_more_books_in_my/",
"https://www.jkrowling.com/news/"
]
}
}
```
Your requested answer, formatted according to the `json` parameter, is in `response.result.json_content` and the list of sources in `response.result.sources`. You can parse the stringified JSON to access the structured data.
```json theme={null}
{
"book_title": "The Hallmarked Man",
"author": "J.K. Rowling",
"release_date": "September 2, 2025"
}
```
Sources example:
```json theme={null}
[
"https://www.harrypotter.com/writing-by-jk-rowling",
"https://stories.jkrowling.com/book-news/",
"https://deadline.com/2024/09/jk-rowling-writing-futuristic-novel-1236093909/",
"https://www.reddit.com/r/FantasticBeasts/comments/1cl1shn/jk_rowling_may_2024_ive_got_six_more_books_in_my/",
"https://www.jkrowling.com/news/"
]
```
When you don't pass the `json` parameter, the API will return a json object with the answer text inside the `result` field.
```json theme={null}
{
"result": "The latest book by J.K. Rowling is The Hallmarked Man."
}
```
### Flexible `json` parameter
* Provide a JSON object with empty values as a schema, or a string describing the data you want.
* If the agent isn’t confident, it returns `NOT_FOUND` for that field.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
answer = client.answers.create(
task="When will React 30 be released?",
json_format={"release_date": ""},
)
print(answer.json_content)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const answer = await client.answers.create({
task: 'When will React 30 be released?',
jsonFormat: { release_date: '' },
})
console.log(answer.json_content)
```
```python Python (API) theme={null}
import requests, json
endpoint = "https://api.olostep.com/v1/answers"
payload = {
"task": "When will React 30 be released?",
"json": {
"release_date": ""
}
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
response = requests.post(endpoint, json=payload, headers=headers)
print(json.dumps(response.json(), indent=2))
```
```bash CLI theme={null}
olostep answer "When will React 30 be released?" --json-format '{"release_date":""}'
```
This would return:
```json theme={null}
{
"release_date": "NOT_FOUND"
}
```
## Pricing
Answers costs 20 credits per request.
# Batch Endpoint
Source: https://docs.olostep.com/features/batches
Best for large-scale, async data aggregation
Through the Olostep `/v1/batches` endpoint you can process up to 10k URLs in one single batch. A batch takes around 5-8 mins. Use it to extract content or structured data at scale in an async way.
* Submit up to 10k URLs per batch
* A single batch takes around 5–8 minutes, regardless of batch size. Batch is an async endpoint
* Run many batches in parallel to scale to millions of concurrent requests.
* Use parsers to return structured JSON, or retrieve markdown/html via `/v1/retrieve`
* If you want to get results with low latency or in a sync way, use the scrape endpoint and send many concurrent requests instead.
For API details see the [Batch Endpoint API Reference](/api-reference/batches/create).
**Note**: For new accounts, batches are limited to 100 items per batch. To lift this limitation, please contact us at [info@olostep.com](mailto:info@olostep.com) or reach out on [Slack](https://olostep-users.slack.com/join/shared_invite/zt-2bfddyi8h-JzfjOgavg~98DJ1om1B5Lg#/shared-invite/email).
## Installation
```python Python theme={null}
pip install olostep
```
```javascript Node theme={null}
npm install olostep
```
```bash cURL theme={null}
# curl is available by default on macOS, Linux, and Windows
```
```javascript Node (API) theme={null}
npm install node-fetch
```
```bash Python (API) theme={null}
pip install requests
```
## Start a batch
Provide an array of `items` with a `custom_id` and `url`. These are the URLs that will be processed in the batch, the `custom_id` is an internal unique identifier for the URL.
Optionally pass `parser` or `country`. Through the `parser` parameter you can specify the parser to use for the batch, this will return structured JSON from the pages.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
batch = client.batches.create(
urls=[
{"custom_id": "item-1", "url": "https://www.google.com/search?q=stripe&gl=us&hl=en"},
{"custom_id": "item-2", "url": "https://www.google.com/search?q=paddle&gl=us&hl=en"},
],
parser="@olostep/google-search",
)
print(batch.id, batch.status)
# Wait and iterate results (auto-waits for completion)
for item in batch.items():
content = item.retrieve(["json"])
print(item.url, item.custom_id)
print(content.json_content)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const batch = await client.batches.create([
{ url: 'https://www.google.com/search?q=stripe&gl=us&hl=en', customId: 'item-1' },
{ url: 'https://www.google.com/search?q=paddle&gl=us&hl=en', customId: 'item-2' },
], {
parser: '@olostep/google-search',
})
console.log(batch.id, batch.total_urls)
// Wait and iterate results (auto-waits for completion)
for await (const item of batch.items()) {
const content = await item.retrieve(['json'])
console.log(item.url, item.custom_id)
console.log(content.json_content)
}
```
```python Python (API) theme={null}
import requests
import hashlib
import time
API_KEY = "" # Replace with your actual API key
API_URL = "https://api.olostep.com/v1"
# Step 1: Utilities
def create_hash_id(url):
return hashlib.sha256(url.encode()).hexdigest()[:16]
def compose_items_array():
urls = [
"https://www.google.com/search?q=ecommerce+platform&gl=us&hl=en",
"https://www.google.com/search?q=payment+gateway&gl=us&hl=en",
"https://www.google.com/search?q=stripe&gl=us&hl=en",
"https://www.google.com/search?q=paddle&gl=us&hl=en",
"https://www.google.com/search?q=merchant+of+record&gl=us&hl=en",
"https://www.google.com/search?q=saas+payments&gl=us&hl=en",
"https://www.google.com/search?q=digital+river&gl=us&hl=en",
"https://www.google.com/search?q=subscription+billing&gl=us&hl=en",
"https://www.google.com/search?q=online+payments&gl=us&hl=en",
"https://www.google.com/search?q=braintree&gl=us&hl=en"
]
# Add the parser configuration to each item
items = []
for url in urls:
items.append({
"custom_id": create_hash_id(url),
"url": url,
})
return items
def start_batch(items):
payload = {
"items": items,
"parser": {"id": "@olostep/google-search"}
}
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
response = requests.post(f"{API_URL}/batches", headers=headers, json=payload)
response.raise_for_status()
return response.json()["id"]
# Step 3: Wait for completion
def check_batch_status(batch_id):
headers = {"Authorization": f"Bearer {API_KEY}"}
response = requests.get(
f"{API_URL}/batches/{batch_id}",
headers=headers
)
response.raise_for_status()
return response.json()["status"]
def wait_until_complete(batch_id):
print("Waiting for batch to complete...")
while True:
status = check_batch_status(batch_id)
print("Status:", status)
if status == "completed":
print("Batch completed!")
return
time.sleep(10)
# Step 4: Get items
def get_completed_items(batch_id):
headers = {"Authorization": f"Bearer {API_KEY}"}
response = requests.get(f"{API_URL}/batches/{batch_id}/items", headers=headers)
response.raise_for_status()
return response.json()["items"]
# Step 5: Retrieve content with format specified
def retrieve_content(retrieve_id):
url = f"{API_URL}/retrieve"
headers = {"Authorization": f"Bearer {API_KEY}"}
params = {
"retrieve_id": retrieve_id,
"formats": ["markdown", "json"]
}
response = requests.get(url, headers=headers, params=params)
response.raise_for_status()
return response.json()
# Step 6: Run end-to-end flow
if __name__ == "__main__":
print("Composing batch...")
items = compose_items_array()
print("Starting batch...")
batch_id = start_batch(items)
print("Batch ID:", batch_id)
print(f"You can check the status of the batch at: {API_URL}/batches/{batch_id}?token={API_KEY}")
wait_until_complete(batch_id)
print("Fetching completed items...")
completed_items = get_completed_items(batch_id)
print("Retrieving content...")
for item in completed_items:
retrieve_id = item["retrieve_id"]
print(f"\nRetrieving content for item {item['retrieve_id']}...")
content = retrieve_content(retrieve_id)
print(f"\n---\nURL: {item['url']}\nCustom ID: {item['custom_id']}\n")
#print("Markdown:\n", content.get("markdown_content", "[No markdown found]"))
print("JSON:\n", content.get("json_content", "[No JSON found]"))
# If you want to see the parsed content specifically
if "parsed_content" in content:
print("\nParsed Content:")
print(content["parsed_content"])
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/batches" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"items": [
{ "custom_id": "item-1", "url": "https://www.google.com/search?q=stripe&gl=us&hl=en" },
{ "custom_id": "item-2", "url": "https://www.google.com/search?q=paddle&gl=us&hl=en" }
],
"parser": { "id": "@olostep/google-search" }
}'
```
```bash CLI theme={null}
# Put your URLs in a CSV with columns: custom_id,url
# Then run:
olostep batch-scrape urls.csv \
--formats json \
--parser-id "@olostep/google-search"
```
```js Node (API) theme={null}
const API_URL = 'https://api.olostep.com/v1'
const payload = {
items: [
{ custom_id: 'item-1', url: 'https://www.google.com/search?q=stripe&gl=us&hl=en' },
{ custom_id: 'item-2', url: 'https://www.google.com/search?q=paddle&gl=us&hl=en' },
],
parser: { id: '@olostep/google-search' }
}
const res = await fetch(`${API_URL}/batches`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify(payload)
})
console.log(await res.json())
```
Since Olostep follows an object-oriented approach, you will receive a `batch` object in response. The `batch` object has a few properties like `id` and `status`.
```json theme={null}
{
"id": "batch_z7n7hwh45x",
"object": "batch",
"status": "in_progress",
"created": 1760329882951,
"total_urls": 10,
"completed_urls": 0,
"number_retried": 0,
"batch_parser": "@olostep/google-search",
"batch_country": "RANDOM",
"start_date": "2025-10-12"
}
```
## Check batch status
Poll until `status` is `completed`. You can also check the `completed_urls` property to see how many URLs have been processed.
```python Python theme={null}
# Using the batch object from the previous step
info = batch.info()
print(info.status, info.completed_urls, info.total_urls)
# Or wait until completed
batch.wait_till_done(check_every_n_secs=10)
```
```js Node theme={null}
// Using the batch object from the previous step
const info = await batch.info()
console.log(info.status, info.completed_urls, info.total_urls)
// Or wait until completed
await batch.waitTillDone({ checkEveryNSecs: 10 })
```
```bash cURL theme={null}
curl -s -X GET "https://api.olostep.com/v1/batches/" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
```js Node (API) theme={null}
const batchId = ''
const info = await fetch(`${API_URL}/batches/${batchId}`, {
headers: { 'Authorization': 'Bearer ' }
}).then(r => r.json())
console.log(info.status)
```
```python Python (API) theme={null}
def get_batch_info(batch_id):
return requests.get(
f"{API_URL}/batches/{batch_id}",
headers={
"Authorization": f"Bearer {API_KEY}"
}
).json()
batch_id = ''
info = get_batch_info(batch_id)
print(info['status'])
```
## Retrieve content
Use the `retrieve_id` from each item with `/v1/retrieve` to fetch `html_content`, `markdown_content`, or `json_content`.
```python Python theme={null}
# Retrieve content for each batch item
for item in batch.items():
content = item.retrieve(["json"])
print(item.url, item.custom_id)
print(content.json_content)
```
```js Node theme={null}
// Retrieve content for each batch item
for await (const item of batch.items()) {
const content = await item.retrieve(['json'])
console.log(item.url, item.custom_id)
console.log(content.json_content)
}
```
```bash cURL theme={null}
curl -s -G "https://api.olostep.com/v1/retrieve" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
--data-urlencode "retrieve_id="
```
```bash CLI theme={null}
# `olostep batch-scrape` retrieves all items automatically. To fetch
# a single item by its retrieve_id later:
olostep scrape-get
```
```js Node (API) theme={null}
const content = await fetch(`${API_URL}/retrieve?retrieve_id=`, {
headers: { 'Authorization': 'Bearer ' }
}).then(r => r.json())
console.log(content.json_content)
```
```python Python (API) theme={null}
def retrieve_content(retrieve_id):
return requests.get(f"{API_URL}/retrieve", headers={"Authorization": f"Bearer {API_KEY}"}, params={"retrieve_id": retrieve_id}).json()
items = list_items('', limit=5)['items']
for item in items:
content = retrieve_content(item['retrieve_id'])
print(content.get('json_content'))
```
## List items (paginate with cursor)
Fetch items using `cursor` and `limit`. Prefer using `/v1/retrieve` with `retrieve_id` for content.
```python Python theme={null}
# Iterate all items (auto-waits for completion, handles pagination)
for item in batch.items():
print(item.custom_id, item.url, item.retrieve_id)
```
```js Node theme={null}
// Iterate all items (auto-waits for completion, handles pagination)
for await (const item of batch.items()) {
console.log(item.custom_id, item.url, item.retrieve_id)
}
```
```bash cURL theme={null}
curl -s -G "https://api.olostep.com/v1/batches//items" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
--data-urlencode "cursor=0" \
--data-urlencode "limit=10"
```
```js Node (API) theme={null}
let cursor = 0
while (true) {
const result = await fetch(`${API_URL}/batches//items?cursor=${cursor}&limit=10`, {
headers: { 'Authorization': 'Bearer ' }
}).then(r => r.json())
result.items.forEach(i => console.log(i.custom_id, i.url, i.retrieve_id))
if (result.cursor === undefined) break
cursor = result.cursor
}
```
```python Python (API) theme={null}
def list_items(batch_id, cursor=None, limit=10):
params = { 'cursor': cursor, 'limit': limit }
return requests.get(f"{API_URL}/batches/{batch_id}/items", headers={"Authorization": f"Bearer {API_KEY}"}, params=params).json()
cursor = 0
while True:
result = list_items('', cursor=cursor, limit=10)
for item in result['items']:
print(item['custom_id'], item['url'], item['retrieve_id'])
if 'cursor' not in result:
break
cursor = result['cursor']
```
## Response Format
When you run the provided example code, you will receive a response like the following
```json theme={null}
Waiting for batch to complete...
Status: in_progress
Status: completed
Batch completed!
Fetching completed items...
Retrieving content...
Retrieving content for item 4pjtky4don_86b1f04cec364903...
---
URL: https://www.google.com/search?q=ecommerce+platform&gl=us&hl=en
Custom ID: 86b1f04cec364903
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"ecommerce platform"},"knowledgeGraph":{"description":"Dec 6, 2024 — To help your business make the right decision, we compare top ecommerce platforms and provide key steps for finding the perfect solution."},"organic":[{"title":"9 Best Ecommerce Platforms of 2025 (Know Your Options)","link":"https://www.bigcommerce.com/articles/ecommerce/ecommerce-platforms/","position":1,"snippet":"Dec 6, 2024 — To help your business make the right decision, we compare top ecommerce platforms and provide key steps for finding the perfect solution."},{"title":"Top 10 Ecommerce Sites in India","link":"https://ecommerceguide.com/top/top-10-ecommerce-sites-in-india/","position":2},{"title":"11 Best Ecommerce Platforms for Your Business in 2025","link":"https://www.shopify.com/blog/best-ecommerce-platforms","position":3,"snippet":"The 11 best ecommerce platforms. Shopify; Wix; BigCommerce; Adobe Commerce; WooCommerce; Squarespace; Big Cartel; Square Online; Shift4Shop; Volusion; OpenCart ..."},{"title":"What is an Ecommerce Platform?","link":"https://www.salesforce.com/commerce/ecommerce-platform/","position":4,"snippet":"An ecommerce platform is software that helps businesses set up and run an online store. It handles product displays, payment processing, order management, and ..."},{"title":"10 Best E-Commerce Platforms Of 2025","link":"https://www.forbes.com/advisor/business/software/best-ecommerce-platform/","position":5,"snippet":"Dec 19, 2024 — 10 Best E-Commerce Platforms Of 2025 · Ecwid · Shift4Shop · Squarespace · BigCommerce · Web.com · Shopify · OpenCart · Big Cartel ..."},{"title":"#1 Ecommerce Shopping Cart & Online Store - Try Ecwid!","link":"https://www.ecwid.com/","position":6,"snippet":"Control everything from a single platform with centralized inventory, order management, and pricing. Get started. Sell everywhere Sell on ..."},{"title":"WooCommerce","link":"https://woocommerce.com/","position":7,"snippet":"WooCommerce empowers you to build, sell, and grow on your terms. Our WordPress-powered platform offers fully customizable ecommerce, for less."},{"title":"eCommerce Website Builder: Build An eCommerce Site","link":"https://www.wix.com/ecommerce/website","position":8,"snippet":"Wix eCommerce is your all-in-one eCommerce platform. Create a fully customizable free eCommerce website & upgrade to start selling everywhere."},{"title":"What E-Commerce platform do you use and why?","link":"https://www.reddit.com/r/ecommerce/comments/1cwo1uo/what_ecommerce_platform_do_you_use_and_why/","position":9,"snippet":"Shopify is hands down one of the best e-commerce platforms. Shopify has a user-friendly interface that simplifies store setup and management."}],"peopleAlsoAsk":[{"question":"What is an ecommerce platform?"},{"question":"What is the best ecommerce platform?"},{"question":"What are the top 5 e-commerce websites?","link":"https://ecommerceguide.com/top/top-10-ecommerce-sites-in-india/","title":"Top 10 Ecommerce Sites in India"},{"question":"What are the 3 types of e-commerce?"}],"relatedSearches":[{"query":"Ecommerce platform examples"},{"query":"e-commerce platforms list"},{"query":"Best ecommerce platform"},{"query":"Best ecommerce platform for beginners"},{"query":"Cheapest ecommerce platform"},{"query":"Best ecommerce platform for small business"},{"query":"Is Amazon an ecommerce platform"},{"query":"Best ecommerce platform for clothing"}]}
Retrieving content for item 4pjtky4don_5d0195299a727a71...
---
URL: https://www.google.com/search?q=payment+gateway&gl=us&hl=en
Custom ID: 5d0195299a727a71
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"payment gateway"},"knowledgeGraph":{"title":"Payment gateway","description":"A payment gateway is a merchant service provided by an e-commerce application service provider that authorizes credit card or direct payment processing for e-businesses, online retailers, bricks and clicks, or traditional brick and mortar.","imageUrl":"https://www.investopedia.com/terms/p/payment-gateway.asp"},"organic":[{"title":"What Is a Payment Gateway? How It Works and Example","link":"https://www.investopedia.com/terms/p/payment-gateway.asp","position":1,"snippet":"A payment gateway is the front-end technology that reads payment cards and sends customer information to the merchant acquiring bank for processing."},{"title":"What are payment gateways? - Stripe","link":"https://stripe.com/resources/more/payment-gateways-101#:~:text=A%20payment%20gateway%20is%20a,a%20secure%20and%20efficient%20manner.","position":2},{"title":"Online payment gateways bring these 5 benefits. - Binary Stream","link":"https://binarystream.com/online-payment-gateways-bring-these-5-benefits/","position":3},{"title":"What are payment gateways?","link":"https://stripe.com/resources/more/payment-gateways-101","position":4,"snippet":"Oct 16, 2023 — A payment gateway is a technology platform that acts as an intermediary in electronic financial transactions. It enables in-person and online ..."},{"title":"9 Best Payment Gateways Of 2025","link":"https://www.forbes.com/advisor/business/software/best-payment-gateways/","position":5,"snippet":"Jan 28, 2025 — 9 Best Payment Gateways Of 2025 · Helcim · PayPal · Shopify · Square · Payline · Clover · Authorize.net · Stripe; Stax. Table of Contents. Table ..."},{"title":"Payment processing: Accept payments anywhere | Authorize.net","link":"https://www.authorize.net/","position":6,"snippet":"Dec 16, 2024 — Accept credit cards, contactless payments, and eChecks in person and on the go. Contact us to learn more by calling 1-888-323-4289."},{"title":"Payment Gateways in 2025: Main Types + How They Work","link":"https://www.bigcommerce.com/articles/ecommerce/payment-gateways/","position":7,"snippet":"Payment gateways are a merchant service that processes credit card payments for both ecommerce sites and traditional brick-and-mortar stores. They can be ..."},{"title":"Payment Gateway, Merchant Services, B2B AR Automation ...","link":"https://paytrace.net/","position":8,"snippet":"Accept payments anywhere with our flexible payment platform. We make it simple and seamless for merchants to accept payments online, in person, and on the go."},{"title":"What Is A Payment Gateway And Why Do I Need One?","link":"https://corporate.freedompay.com/resources/blogs/what-is-a-payment-gateway-and-why-do-i-need-one","position":9,"snippet":"A payment gateway is a software program that sits between the merchant and their customer. It's often described as 'an electronic cash register for the virtual ..."},{"title":"What is a Payment Gateway and how does it work?","link":"https://gocardless.com/en-us/guides/posts/payment-gateways/","position":10,"snippet":"A payment gateway is a tool that securely validates your customer's credit card details, ensuring funds are available for you to get paid."}],"peopleAlsoAsk":[{"question":"What is a payment gateway?","link":"https://stripe.com/resources/more/payment-gateways-101#:~:text=A%20payment%20gateway%20is%20a,a%20secure%20and%20efficient%20manner.","title":"What are payment gateways? - Stripe"},{"question":"What's the best payment gateway?"},{"question":"Is PayPal a payment gateway?"},{"question":"Is Zelle a payment gateway?","link":"https://www.fiserv.com/en/lp/how-zelle-works.html#:~:text=Zelle%20is%20a%20P2P%20payments,credit%20union%20that%20offers%20Zelle.","title":"How Zelle Works - Fiserv"}],"relatedSearches":[{"query":"Payment gateway price"},{"query":"Payment gateway list"},{"query":"Payment gateway PayPal"},{"query":"Payment gateway login"},{"query":"Payment gateway vs payment processor"},{"query":"Types of payment gateway"},{"query":"Payment gateway for website"},{"query":"Payment gateway free"}]}
Retrieving content for item 4pjtky4don_f42462f1e5bb954d...
---
URL: https://www.google.com/search?q=braintree&gl=us&hl=en
Custom ID: f42462f1e5bb954d
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"braintree"},"knowledgeGraph":{"title":"Braintree","type":"Company","description":"PayPal Braintree is a global payment processing solution that delivers end-to-end checkout experiences for businesses, offering single-touch payments, mobile ...","imageUrl":"https://commons.wikimedia.org/wiki/File:Braintree_logo.svg","attributes":{"Founder":"Bryan Johnson","Parent organization":"PayPal","Founded":"October 22, 2007","Company location":"Chicago, Illinois","Headquarters":"Chicago, Illinois, United States","Number of employees":"500+ (2016)"}},"organic":[{"title":"Braintree | Enterprise Payment Solution","link":"https://www.paypal.com/us/braintree","position":1,"snippet":"PayPal Braintree is a global payment processing solution that delivers end-to-end checkout experiences for businesses, offering single-touch payments, mobile ..."},{"title":"Braintree vs. Paypal: Which is Better for Payment Processing - Tipalti","link":"https://tipalti.com/resources/learn/braintree-vs-paypal/#:~:text=Braintree%20offers%20more%20fraud%20protection,cryptocurrency%20buy%20and%20sell%20transactions.","position":2},{"title":"Braintree, MA | Official Website","link":"https://www.braintreema.gov/","position":3,"snippet":"Braintree Town Hall ... Town Hall is open Mon, Wed, Thur 8:30 AM to 4:30 PM, Tue 8:30 AM - 7:00 PM, Fri 8:30 AM - 1:00 PM."},{"title":"Braintree (company)","link":"https://en.wikipedia.org/wiki/Braintree_(company)","position":4,"snippet":"Braintree is a Chicago-based company that primarily deals in mobile and web payment systems for e-commerce companies. The company was acquired by PayPal on ..."},{"title":"Braintree Payment Gateway: Features, Pricing And Reviews","link":"https://staxpayments.com/blog/braintree-payment-gateway/#:~:text=Braintree%20offers%20transparent%20pricing%20with%20a%20fee%20of%202.59%25%20plus,2.9%25%20plus%20%240.30%20per%20transaction.","position":5},{"title":"Braintree Scientific","link":"https://www.braintreesci.com/","position":6,"snippet":"We are a women owned small business, dedicated to providing new product technology and supplies to advance medical milestones for all. You've seen our unique ..."},{"title":"HOME | Braintree Academy","link":"https://www.braintree4me.com/","position":7,"snippet":"Braintree Academy is a tuition-free, virtual public school program tailored to the unique needs of each student and family."},{"title":"Braintree, Massachusetts","link":"https://en.wikipedia.org/wiki/Braintree,_Massachusetts","position":8,"snippet":"Braintree is a municipality in Norfolk County, Massachusetts, United States. It is officially known as a town, but Braintree is a city with a mayor-council ..."},{"title":"BrainTree Nutrition Collection | Brain Health Supplements","link":"https://www.braintreenutrition.com/collections/all?srsltid=AfmBOor3eehM1T3xOqtn2i1wApxaWrFtRMoOCp6en_4X24_JXoYoWptj","position":9,"snippet":"We offer the finest nutritional products for cognitive enhancement, brain health and anti-aging. Fortify long-lasting brain health and unrivaled cognitive power ..."},{"title":"Braintree Gateway Login","link":"https://www.braintreegateway.com/login","position":10,"snippet":"A username can be either: A unique identifier designated by you at signup (e.g. yourusername100); The email you signed up with (e.g. example@email.com)."},{"title":"PayPal Braintree Fees & Pricing","link":"https://www.paypal.com/us/enterprise/paypal-braintree-fees","position":11,"snippet":"Custom flat rates, interchange plus pricing, and discounted rates are available for established businesses based on business model and processing volume."}],"peopleAlsoAsk":[{"question":"What is Braintree used for?"},{"question":"How much did Bryan Johnson sell Braintree for?"},{"question":"What is the difference between PayPal and Braintree?","link":"https://tipalti.com/resources/learn/braintree-vs-paypal/#:~:text=Braintree%20offers%20more%20fraud%20protection,cryptocurrency%20buy%20and%20sell%20transactions.","title":"Braintree vs. Paypal: Which is Better for Payment Processing - Tipalti"},{"question":"Is Braintree the same as Venmo?","link":"https://en.wikipedia.org/wiki/Braintree_(company)#:~:text=In%202012%2C%20Braintree%20acquired%20Venmo%20for%20%2426.2%20million.","title":"Braintree (company) - Wikipedia"}],"relatedSearches":[{"query":"Braintree pricing"},{"query":"Braintree login"},{"query":"Braintree PayPal"},{"query":"Braintree payments"},{"query":"Braintree school"},{"query":"Braintree company"},{"query":"Braintree sandbox"},{"query":"Braintree city"}]}
Retrieving content for item 4pjtky4don_256e5f8ec1074fca...
---
URL: https://www.google.com/search?q=digital+river&gl=us&hl=en
Custom ID: 256e5f8ec1074fca
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"digital river"},"knowledgeGraph":{"title":"Digital River","type":"E-commerce company","description":"Digital River, Inc. was a privately held company headquartered in Minnetonka, Minnesota that provides global e-commerce, payments and marketing services.","imageUrl":"https://www.digitalriver.com/","attributes":{"Headquarters":"Minnetonka, MN","CEO":"Adam Coyle (Jul 10, 2018–)","Founder":"Joel A. Ronning","Founded":"February 1994","Number of employees":"122 (2025)","Traded as":"Nasdaq: DRIV (1998–2015)"}},"organic":[{"title":"Digital River","link":"https://en.wikipedia.org/wiki/Digital_River","position":1,"snippet":"Digital River, Inc. was a privately held company headquartered in Minnetonka, Minnesota that provides global e-commerce, payments and marketing services."},{"title":"US paytech Digital River reportedly shutting down","link":"https://www.fintechfutures.com/2025/02/us-paytech-digital-river-reportedly-shutting-down/#:~:text=Founded%20in%201994%2C%20Digital%20River,%2C%20Salesforce%2C%20and%20American%20Express.","position":2},{"title":"US paytech Digital River reportedly shutting down","link":"https://www.fintechfutures.com/digital-payments/us-paytech-digital-river-reportedly-shutting-down","position":3,"snippet":"Feb 11, 2025 — US-based payments firm Digital River is reportedly winding down its operations, according to local media outlet The Minnesota Star Tribune."},{"title":"Minnetonka-based E-Commerce Firm Digital River to Shut ...","link":"https://tcbmag.com/minnetonka-based-e-commerce-firm-digital-river-to-shut-down/","position":4,"snippet":"Jan 28, 2025 — Minnetonka-based e-commerce firm Digital River is laying off 122 employees locally and winding down global operations elsewhere."},{"title":"Renew Digital River subscriptions directly with Adobe","link":"https://helpx.adobe.com/x-productkb/policy-pricing/digital-river-deprecation-adobe.html#:~:text=Why%20can't%20I%20get%20a%20refund%20for%20my%20purchase,any%20inconvenience%20this%20may%20cause.","position":5},{"title":"I have a \"Digital River\" transaction on my bank statement ...","link":"https://www.reddit.com/r/personalfinance/comments/daj103/i_have_a_digital_river_transaction_on_my_bank/","position":6,"snippet":"Digital River is a legitimate e-commerce vendor. They help businesses setup their online stores. Have you made any online purchases recently?"},{"title":"Digital River (@DigitalRiverInc) / X","link":"https://x.com/digitalriverinc?lang=en","position":7,"snippet":"Jul 2, 2024 — The ultimate ecommerce accelerator for global growth. Fast, easy, risk-free expansion into 240+ destinations. Accelerate. Simplify. Optimize."},{"title":"Digital River","link":"https://www.linkedin.com/company/digital-river","position":8,"snippet":"We're proactive partners, providing API-based Cross-Border, Order Management and Ecommerce services to leading enterprise brands.","meta":"68.9K+ followers"},{"title":"Digital River Insolvency: A Guide for Affected Software ...","link":"https://freemius.com/blog/digital-river-mycommerce-shutdown-guide/","position":9,"snippet":"Jan 31, 2025 — Digital River abruptly laid off over 100 employees and filed for insolvency, leaving software creators in the lurch, unable to access their hard-earned revenue."},{"title":"Digital River cuts staff and will close headquarters","link":"https://www.digitalcommerce360.com/2025/01/29/digital-river-cuts-staff-will-shut-down-headquarters/","position":10,"snippet":"Jan 29, 2025 — The company will close its Minnesota headquarters by the end of March, impacting 122 employees, including remote workers nationwide."},{"title":"Connecting with Digital River","link":"https://support.bigcommerce.com/s/article/Connecting-with-Digital-River","position":11,"snippet":"Digital River is a global online payments platform engineered to maximize conversions and grow revenue. Digital River partners with the world's leading ..."},{"title":"Who is Digital River Ireland, Ltd.","link":"https://www.paypal-community.com/t5/Security-and-Fraud/Who-is-Digital-River-Ireland-Ltd/td-p/3055369","position":12,"snippet":"Apr 17, 2023 — Digital river is an ecommerce payment company. They don't fraudulently take your money. They partner with companies like Samsung, Microsoft, ..."}],"peopleAlsoAsk":[{"question":"Is Digital River still in business?"},{"question":"What is the Digital River?","link":"https://www.fintechfutures.com/2025/02/us-paytech-digital-river-reportedly-shutting-down/#:~:text=Founded%20in%201994%2C%20Digital%20River,%2C%20Salesforce%2C%20and%20American%20Express.","title":"US paytech Digital River reportedly shutting down"},{"question":"How do I stop my Digital River subscription?","link":"https://emma-app.com/how-to-cancel-digital-river#:~:text=To%20cancel%20a%20Digital%20River,contact%20their%20customer%20service%20there.","title":"How To Cancel Digital River - Emma app"},{"question":"What is happening at Digital River?","link":"https://gappgroup.com/blog/digital-river-is-shutting-down-heres-how-gapp-group-can-seamlessly-support-its-customers/#:~:text=The%20eCommerce%20industry%20was%20recently,before%20their%20operations%20are%20disrupted.","title":"Digital River Is Shutting Down – Here's How Gapp Group Can ..."}],"relatedSearches":[{"query":"Digital River payment"},{"query":"Digital River charge on credit card"},{"query":"Digital River subscription"},{"query":"Digital river complaints"},{"query":"Digital river app"},{"query":"Digital River Adobe"},{"query":"Digital River news"},{"query":"Digital River NVIDIA"}]}
Retrieving content for item 4pjtky4don_625d3a44aeb9123b...
---
URL: https://www.google.com/search?q=online+payments&gl=us&hl=en
Custom ID: 625d3a44aeb9123b
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"online payments"},"knowledgeGraph":{"description":"Pay your income tax, property tax, college tuition, utility and other bills online with a credit card, debit card or other convenient option."},"organic":[{"title":"ACI Payments, Inc. - Pay Taxes, Utility Bills, Tuition & More ...","link":"https://www.officialpayments.com/","position":1,"snippet":"Pay your income tax, property tax, college tuition, utility and other bills online with a credit card, debit card or other convenient option."},{"title":"The 6 best online payment processing services in 2025 - Zapier","link":"https://zapier.com/blog/best-payment-gateways/","position":2},{"title":"Pay, Send and Save Money with PayPal | PayPal US","link":"https://www.paypal.com/","position":3,"snippet":"Tap to pay safely in stores with the PayPal Debit Card and earn rewards online with PayPal checkout. Get even more cash back on the brands you love."},{"title":"Online Payments Made Simple | Pay.com","link":"https://pay.com/","position":4,"snippet":"Pay.com lets you easily accept online payments and grow your revenue. Start accepting credit and debit cards, digital wallets, and other payment methods."},{"title":"Payments | Internal Revenue Service","link":"https://www.irs.gov/payments","position":5,"snippet":"View amount due, payment plan details, payment history and scheduled payments; Pay separate assessment payments. Pay in online account. Business Tax Account."},{"title":"Seamlessly Pay Online, Pay In Stores or Send Money","link":"https://pay.google.com/about/","position":6,"snippet":"Google Pay is a quick, easy, and secure way to pay online, in stores or send money to friends and family. Pay the Google way."},{"title":"Stripe | Financial Infrastructure to Grow Your Revenue","link":"https://stripe.com/","position":7,"snippet":"Stripe powers online and in-person payment processing and financial solutions for businesses of all sizes. Accept payments, send payouts, and automate ..."},{"title":"Collecting online payments: How It works","link":"https://stripe.com/guides/introduction-to-online-payments","position":8,"snippet":"This guide offers a high-level overview of online payments and covers nuances based on different business models. Learn more."},{"title":"The Best Online Payment Service Providers in 2023","link":"https://www.wildapricot.com/blog/online-payment-services","position":9,"snippet":"Jul 14, 2023 — The Top 11 Best Online Payment Systems For Your Organization · 1. WildApricot Payments · 2. Stripe · 3. Apple Pay · 4. Dwolla · 5. Due · 6."},{"title":"Pay.gov - Home","link":"https://www.pay.gov/","position":10,"snippet":"Pay an overdue debt to the Bureau of the Fiscal Service. Do you want to make a payment toward a federal non-tax debt (not an IRS tax debt or student loan debt)?."}],"peopleAlsoAsk":[{"question":"Which site is best for online payments?","link":"https://zapier.com/blog/best-payment-gateways/","title":"The 6 best online payment processing services in 2025 - Zapier"},{"question":"Which is best for online payments?","link":"https://www.dsgpay.com/blog/digital-payment-services-in-india/","title":"10 Best Digital Payment Services in India 2024: Pros and Cons - DSGPay"},{"question":"What is meant by online payment?","link":"https://www.pinelabs.com/blog/online-payments-and-its-types-methods-and-meaning#:~:text=Online%20payment%20allows%20you%20to,net%20banking%2C%20and%20digital%20wallets.","title":"What is Online Payment? Types, Modes, Methods, Meaning"},{"question":"What is the best way to pay online?"}],"relatedSearches":[{"query":"Free online payments"},{"query":"Online payments credit card"},{"query":"IRS payment online"},{"query":"Online payments app"},{"query":"ACI Payments online"},{"query":"Pay estimated taxes online"},{"query":"ACI pay online Customer Service"},{"query":"PayPal"}]}
Retrieving content for item 4pjtky4don_bd045d49c210b22a...
---
URL: https://www.google.com/search?q=stripe&gl=us&hl=en
Custom ID: bd045d49c210b22a
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"stripe"},"knowledgeGraph":{"title":"Stripe, Inc.","type":"Financial services company","description":"Reduce costs, grow revenue, and run your business more efficiently on a fully integrated platform. Use Stripe to handle all of your payments-related needs, ...","website":"http://stripe.com/","imageUrl":"http://t3.gstatic.com/images?q=tbn:ANd9GcSJHbnfk81kA_5mIj81yhRy3R2LRx3S11OyMjC68QeONsOp5DXx","attributes":{"Founders":"Patrick Collison, John Collison","Headquarters":"Dublin, Ireland","CEO":"Patrick Collison (2010–)","Revenue":"14.4 billion USD (2022)","Founded":"2010, San Francisco, CA","Number of employees":"8,500 (2025)"}},"organic":[{"title":"Stripe | Financial Infrastructure to Grow Your Revenue","link":"https://stripe.com/","position":1,"snippet":"Reduce costs, grow revenue, and run your business more efficiently on a fully integrated platform. Use Stripe to handle all of your payments-related needs, ...","sitelinks":[{"title":"Login","link":"https://dashboard.stripe.com/login"},{"title":"Create your Stripe account","link":"https://dashboard.stripe.com/register"},{"title":"Jobs","link":"https://stripe.com/jobs"},{"title":"Pricing & Fees","link":"https://stripe.com/pricing"},{"title":"Support","link":"https://support.stripe.com/"}]},{"title":"What Is Stripe, and How Does It Work to Accept Payments? - NerdWallet","link":"https://www.nerdwallet.com/article/small-business/what-is-stripe#:~:text=Stripe%20is%20a%20payment%20processing,platform's%20developer%20tools%20and%20customizability.","position":2},{"title":"Stripe - X","link":"https://x.com/stripe?lang=en","position":3,"snippet":"Stripe is a global technology company that builds economic infrastructure for the internet. Help: @stripesupport. Read: @stripepress. Status: @stripestatus."},{"title":"Stripe, Inc.","link":"https://en.wikipedia.org/wiki/Stripe,_Inc.","position":4,"snippet":"Stripe is the largest privately-owned fintech company with a valuation of about $91 billion and over $1.4 trillion in payment volume processed in 2024."},{"title":"Stripe","link":"https://www.linkedin.com/company/stripe","position":5,"snippet":"Stripe is a financial infrastructure platform for businesses. Millions of companies—from the world's largest enterprises to the most ...","meta":"1M+ followers"}],"peopleAlsoAsk":[{"question":"What does Stripe exactly do?","link":"https://www.nerdwallet.com/article/small-business/what-is-stripe#:~:text=Stripe%20is%20a%20payment%20processing,platform's%20developer%20tools%20and%20customizability.","title":"What Is Stripe, and How Does It Work to Accept Payments? - NerdWallet"},{"question":"Is Stripe owned by Elon Musk?","link":"https://en.wikipedia.org/wiki/Stripe,_Inc.#:~:text=In%202011%20the%20company%20received,Andreessen%20Horowitz%2C%20and%20SV%20Angel.","title":"Stripe, Inc. - Wikipedia"},{"question":"What is Stripe and is it legit?","link":"https://www.acodei.com/blog/is-stripe-safe-and-secure#:~:text=Stripe%20is%20a%20highly%20secure%20payment%20platform%20trusted%20by%20businesses%20worldwide.","title":"Is Stripe Safe and Secure? - Acodei Blog"},{"question":"How much is the Stripe fee for $100?"}],"relatedSearches":[{"query":"Stripe fees"},{"query":"Stripe login"},{"query":"Stripe careers"},{"query":"Stripe Express"},{"query":"Stripe sign up"},{"query":"Stripe Dashboard"},{"query":"Stripe payment"},{"query":"Stripe logo"}]}
Retrieving content for item 4pjtky4don_228d8979b098d94d...
---
URL: https://www.google.com/search?q=subscription+billing&gl=us&hl=en
Custom ID: 228d8979b098d94d
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"subscription billing"},"knowledgeGraph":{"description":"Stripe Billing lets you bill and manage customers however you want—from simple recurring billing to usage-based billing and sales-negotiated contracts."},"organic":[{"title":"Stripe Billing | Recurring Payments & Subscription ...","link":"https://stripe.com/billing","position":1,"snippet":"Stripe Billing lets you bill and manage customers however you want—from simple recurring billing to usage-based billing and sales-negotiated contracts."},{"title":"Find your purchases, reservations & subscriptions - Android - Google Help","link":"https://support.google.com/accounts/answer/7673989?hl=en&co=GENIE.Platform%3DAndroid","position":2},{"title":"Subscription billing overview - Finance | Dynamics 365","link":"https://learn.microsoft.com/en-us/dynamics365/finance/accounts-receivable/subscription-billing-summary","position":3,"snippet":"Mar 13, 2024 — Subscription billing enables organizations to manage subscription revenue opportunities and recurring billing through billing schedules."},{"title":"What is Subscription Billing?","link":"https://dealhub.io/glossary/subscription-billing/","position":4,"snippet":"Sep 8, 2024 — Subscription billing is a type of recurring billing in which customers are charged a set amount at regular intervals for a service or product."},{"title":"What is Subscription Billing? A Guide for SaaS Businesses","link":"https://www.younium.com/blog/what-is-subscription-billing","position":5,"snippet":"Subscription billing involves collecting recurring payments from customers automatically at set intervals. With this model, you require your subscribers to sign ..."},{"title":"Recurly: Subscription Management Software & Recurring ...","link":"https://recurly.com/","position":6,"snippet":"Recurly is the best subscription management software and recurring billing platform on the market, compatible with leading ERP, CRM, payment gateways, ..."},{"title":"Billing and Subscriptions","link":"https://support.apple.com/billing","position":7,"snippet":"Manage your payment information. View and update your payment methods or update your billing information. Change, add, or remove a payment method. If you're ..."},{"title":"SAP Subscription Billing | Recurring Payment Management","link":"https://www.sap.com/products/financial-management/subscription-billing.html","position":8,"snippet":"Run simplified, automated billing and ordering processes designed for experience-centric consumers by using the SAP Subscription Billing solution."},{"title":"Recurring payments vs. subscription billing","link":"https://stripe.com/resources/more/recurring-payments-vs-subscription-billing","position":9,"snippet":"Apr 6, 2023 — Subscription billing is a payment model that allows businesses to charge recurring payments for access to products or services. Subscription ..."}],"peopleAlsoAsk":[{"question":"What is subscription billing?"},{"question":"How do I find my list of all my subscriptions?","link":"https://support.google.com/accounts/answer/7673989?hl=en&co=GENIE.Platform%3DAndroid","title":"Find your purchases, reservations & subscriptions - Android - Google Help"},{"question":"What is subscription billing in Mycase?","link":"https://supportcenter.mycase.com/en/articles/9369989-subscription-billing#:~:text=Subscription%20Billing%20allows%20you%20to,for%20a%20recurring%20set%20fee.","title":"Subscription Billing - MyCase Help Center"},{"question":"How to bill a client for a subscription?","link":"https://toggl.com/blog/how-to-bill-clients","title":"How to Bill a Client for the First Time: A Step-by-Step Guide - Toggl Track"}],"relatedSearches":[{"query":"Subscription billing D365"},{"query":"Subscription billing software"},{"query":"Subscription billing cost"},{"query":"Subscription billing - Business Central"},{"query":"Subscription Billing SAP"},{"query":"Subscription billing Patreon"},{"query":"Subscription billing Apple"},{"query":"Payment and subscription Google"}]}
Retrieving content for item 4pjtky4don_961fb5d15f9bb584...
---
URL: https://www.google.com/search?q=paddle&gl=us&hl=en
Custom ID: 961fb5d15f9bb584
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"paddle"},"knowledgeGraph":{"title":"Paddle","description":"A paddle is a handheld tool with an elongated handle and a flat, widened end used as a lever to apply force onto the bladed end. It most commonly describes a completely handheld tool used to propel a human-powered watercraft by pushing water in a direction opposite to the direction of travel.","imageUrl":"https://www.amazon.com/Large-Bamboo-Paddle-Wooden-Airflow/dp/B0C9MH329T"},"organic":[{"title":"Paddle - Payments, tax and subscription management for ...","link":"https://www.paddle.com/","position":1,"snippet":"The Merchant of Record for digital products · 5,000+ SaaS, AI and app businesses use Paddle to solve all their payments, tax and compliance needs. Because the ...","sitelinks":[{"title":"Why has Paddle charged me?","link":"https://www.paddle.com/about/why-has-paddle-charged-me"},{"title":"Subscription Management","link":"https://www.paddle.com/billing/subscriptions"},{"title":"Pricing","link":"https://www.paddle.com/pricing"},{"title":"Get started","link":"https://www.paddle.com/get-started"},{"title":"About us","link":"https://www.paddle.com/about"}]},{"title":"PADDLE Definition & Meaning - Merriam-Webster","link":"https://www.merriam-webster.com/dictionary/paddle#:~:text=a,for%20stirring%2C%20mixing%2C%20or%20hitting","position":2},{"title":"PADDLE Definition & Meaning","link":"https://www.merriam-webster.com/dictionary/paddle","position":3,"snippet":"1. a : a usually wooden implement that has a long handle and a broad flattened blade and that is used to propel and steer a small craft (such as a canoe)"},{"title":"Squash, pickleball, padel, tennis: popular paddle sports, explained","link":"https://www.houstonchronicle.com/neighborhood/woodlands/article/squash-pickleball-padel-tennis-paddle-sports-trend-18444545.php#:~:text=Squash%2C%20pickleball%2C%20padel%2C%20tennis%3A%20popular%20paddle%20sports%2C%20explained","position":4},{"title":"Paddle","link":"https://en.wikipedia.org/wiki/Paddle","position":5,"snippet":"A paddle is a handheld tool with an elongated handle and a flat, widened end (the blade) used as a lever to apply force onto the bladed end."},{"title":"Roc Inflatable Stand Up Paddle Boards with Premium SUP ...","link":"https://www.amazon.com/Outdoors-Roc-Inflatable-Accessories-Non-Slip/dp/B0D3VS2QCJ","position":6,"snippet":"The adjustable paddle is perfect for either SUP or kayaking, and the waterproof dry bag is the perfect bonus--plus you can strap it to the front or back of the board for easy access."},{"title":"Bent Paddle Brewing Company: Home","link":"https://bentpaddlebrewing.com/","position":7,"snippet":"Visit the family friendly Bent Paddle Brewing taproom in the heart of Lincoln Park. Featuring craft beer, hemp bevs, live music + local eats!"},{"title":"paddle","link":"https://en.wiktionary.org/wiki/paddle","position":8,"snippet":"Verb · English 2-syllable words · English terms with IPA pronunciation · English terms with audio pronunciation · Rhymes:English/ædəl · Rhymes:English/ædəl/2 ..."}],"peopleAlsoAsk":[{"question":"What is a paddle?","link":"https://www.merriam-webster.com/dictionary/paddle#:~:text=a,for%20stirring%2C%20mixing%2C%20or%20hitting","title":"PADDLE Definition & Meaning - Merriam-Webster"},{"question":"What is paddle.com charging me for?","link":"https://www.paddle.com/about/why-has-paddle-charged-me#:~:text=If%20you're%20seeing%20a,software%20companies%20in%20our%20network.","title":"Why do I have a charge from Paddle? - FAQs"},{"question":"Is paddle like pickleball?","link":"https://www.ppatour.com/ppa-blog/padel-vs-pickleball/#:~:text=Pickleball%20and%20padel%20are%20both,game%20play%2C%20equipment%20and%20more.","title":"Padel vs Pickleball - PPA Tour"},{"question":"Is it padel or paddle?"}],"relatedSearches":[{"query":"Paddle tennis"},{"query":"Paddle sport"},{"query":"Paddle or padel"},{"query":"Paddle meaning in Hindi"},{"query":"paddle.com market limited"},{"query":"Boat Paddle"},{"query":"Paddle payment"},{"query":"Paddle board"}]}
Retrieving content for item 4pjtky4don_bb93e46ea1d317e7...
---
URL: https://www.google.com/search?q=merchant+of+record&gl=us&hl=en
Custom ID: bb93e46ea1d317e7
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"merchant of record"},"knowledgeGraph":{"description":"Aug 1, 2024 — A merchant of record (MoR) is a legal entity responsible for selling goods or services to an end customer. They manage all payments and take on ..."},"organic":[{"title":"Service (economics) - Wikipedia","link":"https://en.wikipedia.org/wiki/Service_(economics)","position":1},{"title":"What is a merchant of record (MoR) + why use one ... - Paddle","link":"https://www.paddle.com/blog/what-is-merchant-of-record#:~:text=A%20merchant%20of%20record%20(MoR)%20is%20a%20legal%20entity%20responsible,and%20honoring%20refunds%20and%20chargebacks.","position":2},{"title":"What is a merchant of record (MoR) + why use one ...","link":"https://www.paddle.com/blog/what-is-merchant-of-record","position":3,"snippet":"Aug 1, 2024 — A merchant of record (MoR) is a legal entity responsible for selling goods or services to an end customer. They manage all payments and take on ..."},{"title":"What Is a Merchant of Record? (And Why Should You Care?)","link":"https://fastspring.com/blog/what-is-a-merchant-of-record-and-why-you-should-care/","position":4,"snippet":"Sep 24, 2024 — A merchant of record (MoR) is the legal entity that sells goods or services to a customer. Companies can be their own MoR, but you can also ..."},{"title":"Merchant of Record vs. Seller of Record: Key Differences ...","link":"https://passportglobal.com/blog/merchant-of-record-vs-seller-of-record-key-differences-impacts-on-international-ecommerce/","position":5,"snippet":"Feb 19, 2025 — A Merchant of Record (MOR) assumes legal and financial responsibilities in transactions, acting as an intermediary for the brand. In contrast, a ..."},{"title":"Merchant of record payment services feedback : r/SaaS","link":"https://www.reddit.com/r/SaaS/comments/1bcwmyd/merchant_of_record_payment_services_feedback/","position":6,"snippet":"Merchant of Record is practically a reseller, hence you get to benefits (on top of your payments requirements) that a payment provider would not ..."},{"title":"What is a merchant of record?","link":"https://www.checkout.com/blog/what-is-a-merchant-of-record","position":7,"snippet":"Aug 10, 2023 — A merchant of record (MoR) is a professional service that takes responsibility for selling goods or services to an end consumer on behalf of a ..."},{"title":"Merchant of Record Vs Payment Gateway","link":"https://gappgroup.com/blog/merchant-of-record-vs-payment-gateway/","position":8,"snippet":"A Merchant of Record is a legal entity authorized to sell products or services to customers and process their credit and debit card transactions on behalf of ..."},{"title":"What is a Merchant of Record?","link":"https://gocardless.com/guides/posts/what-is-a-merchant-of-record/","position":9,"snippet":"A Merchant of Record is the term to describe a legal entity that handles all payments. In addition, it takes on the liability related to every transaction to an ..."}],"peopleAlsoAsk":[{"question":"What does being a merchant of record mean?","link":"https://www.paddle.com/blog/what-is-merchant-of-record#:~:text=A%20merchant%20of%20record%20(MoR)%20is%20a%20legal%20entity%20responsible,and%20honoring%20refunds%20and%20chargebacks.","title":"What is a merchant of record (MoR) + why use one ... - Paddle"},{"question":"What is the difference between merchant of record and seller of record?"},{"question":"Is Amazon a merchant of record?","link":"https://www.depositfix.com/blog/merchant-of-record#:~:text=Types%20of%20Merchant%20of%20Record&text=It%20conducts%20transactions%20under%20its,an%20MoR%20for%20various%20sellers.","title":"Mastering Merchant of Record: A Comprehensive Business Guide"},{"question":"Is PayPal a merchant of record?"}],"relatedSearches":[{"query":"Merchant of record meaning"},{"query":"Merchant of record examples"},{"query":"Merchant of record Stripe"},{"query":"Merchant of record companies"},{"query":"Merchant of record VAT"},{"query":"Merchant of record Wiki"},{"query":"Merchant of record Shopify"},{"query":"Paddle merchant of record"}]}
Retrieving content for item 4pjtky4don_e7cd78a461046022...
---
URL: https://www.google.com/search?q=saas+payments&gl=us&hl=en
Custom ID: e7cd78a461046022
JSON:
{"searchParameters":{"type":"search","engine":"google","q":"saas payments"},"knowledgeGraph":{"description":"Launch new plans and start accepting payments in minutes. Collect and store payment details including cards, ACH, and other popular payment methods. · Support ..."},"organic":[{"title":"Billing Platform for SaaS Businesses","link":"https://stripe.com/use-cases/saas","position":1,"snippet":"Launch new plans and start accepting payments in minutes. Collect and store payment details including cards, ACH, and other popular payment methods. · Support ..."},{"title":"Guidance for students from Scotland - GOV.UK","link":"https://www.gov.uk/guidance/guidance-for-students-from-scotland#:~:text=Student%20Loans%20Company%20(%20SLC%20)%20pays,do%20not%20pay%20student%20loans.","position":2},{"title":"What payment system do you use for your SaaS?","link":"https://www.reddit.com/r/SaaS/comments/1ejbn9i/what_payment_system_do_you_use_for_your_saas/","position":3,"snippet":"Try depositfix.com. It integrates a lot of possible payment systems, integrates with crm as well. Its great. Upvote 2. Downvote Reply reply"},{"title":"SaaS payment processing 101","link":"https://stripe.com/resources/more/challenges-of-saas-payment-processing","position":4,"snippet":"Mar 9, 2023 — SaaS payment processing refers to the way companies accept and process payments. While ecommerce sales consist of one-off online payments, SaaS ..."},{"title":"Moving Up(front) with Upfront SaaS Payments","link":"https://gsablogs.gsa.gov/technology/2024/07/25/moving-upfront-with-upfront-saas-payments/","position":5,"snippet":"Jul 25, 2024 — To offer the upfront payment option, vendors must submit a modification adding it to their schedule contract. We encourage vendors to offer SaaS ..."},{"title":"Understanding SaaS Payment Processing: Implementation ...","link":"https://staxpayments.com/blog/saas-payment-processing/","position":6,"snippet":"A SaaS payment processor is a service provider that focuses on providing the tools that SaaS companies need to manage payments and subscriptions accurately."},{"title":"Paddle - Payments, tax and subscription management for ...","link":"https://www.paddle.com/","position":7,"snippet":"The only complete billing solution for digital products. Payments, tax, subscription management and more, all handled for you. Discover Billing. SaaS billing ..."},{"title":"SaaS Spend Management Solution","link":"https://meshpayments.com/saas-spend-management/","position":8,"snippet":"Manage all your SaaS subscriptions from one platform, with actionable insights and controls at the right time to continuously optimize your spend."},{"title":"Top SaaS Payment Solutions for Streamlining Transactions","link":"https://wise.com/us/blog/saas-payment-solutions","position":9,"snippet":"Mar 10, 2025 — SaaS payment solutions are specialized payment platforms that enable software providers to efficiently manage and process recurring subscription ..."},{"title":"SaaS payments: everything SaaS businesses need to know","link":"https://www.checkout.com/blog/saas-payments","position":10,"snippet":"Sep 19, 2023 — SaaS payments are charged by subscription businesses on a recurring basis in exchange for access to a service."}],"peopleAlsoAsk":[{"question":"What are SaaS payments?","link":"https://www.gov.uk/guidance/guidance-for-students-from-scotland#:~:text=Student%20Loans%20Company%20(%20SLC%20)%20pays,do%20not%20pay%20student%20loans.","title":"Guidance for students from Scotland - GOV.UK"},{"question":"What are SaaS transactions?","link":"https://www.cognism.com/sales-saas#:~:text=SaaS%20sales%20is%20the%20process,their%20pain%20points%20or%20problems.","title":"What Is SaaS Sales? Everything You Need to Know in 2025 - Cognism"},{"question":"What does SaaS mean?","link":"https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-is-saas#:~:text=Software%20as%20a%20service%20definition,purchasing%20and%20installing%20them%20locally.","title":"What is Software as a Service (SaaS)? - Microsoft Azure"},{"question":"What date is the SaaS payment?","link":"https://www.saas.gov.uk/need-to-know/payments/loan-payments#:~:text=The%20SLC%20will%20issue%20a,released%20at%20the%20same%20time.","title":"Loan Payments - SAAS"}],"relatedSearches":[{"query":"Saas payments login"},{"query":"Saas payments reddit"},{"query":"SaaS billing"},{"query":"SaaS billing software"},{"query":"Best SaaS billing software"},{"query":"B2B SaaS billing software"},{"query":"Stripe SaaS billing"},{"query":"SaaS billing models"}]}
```
Since we passed the parser to retrieve the structured JSON and we are printing only that, the response contains:
* **json\_content** with structured search results including:
* `searchParameters`: Information about the search query
* `knowledgeGraph`: Detailed information about the search subject (when available)
* `organic`: List of search results with title, link, position, and snippet
* `peopleAlsoAsk`: Related questions that users commonly search for
* `relatedSearches`: Suggested related search queries
If you don't want the structured JSON but simply the markdown or html you can retrieve those from the retrieve endpoint.
## Webhooks
Instead of polling batch status, you can pass a **`webhook`** URL when you create the batch. Olostep sends an **HTTP POST** to that URL when the batch finishes (all items completed or failed).
Your webhook endpoint must be **publicly reachable** over **`http://` or `https://`**. It cannot point to localhost or private IP addresses. For the full payload shape, retry behavior, and best practices (respond quickly with `2xx`, deduplicate using the event `id`), see [Webhooks](/api-reference/common/webhooks).
**Parameter name:** The canonical field is `webhook`. For backward compatibility, **`webhook_url`** is also accepted as an alias.
For batches, the **`batch.completed`** event includes the batch id, status, and item counts. Failed deliveries are retried automatically (up to **5 attempts** over about **30 minutes** with exponential backoff). Your handler must return a **2xx** status within **30 seconds** per attempt.
## Metadata
Attach custom **string key-value** metadata to batches for tracking, filtering, and correlating jobs with your own systems (order IDs, project names, pipeline stage, and so on). Metadata follows the same rules as in our [Metadata](/api-reference/common/metadata) reference.
You can set metadata at **two levels** when creating a batch:
* **Batch-level** — `metadata` on the **request body** (applies to the whole batch)
* **Item-level** — `metadata` on **each object** in the `items` array (per URL)
Metadata is **returned** on subsequent **GET** responses for that batch. You can **merge-update** batch metadata later with [Update Batch](/api-reference/batches/update) (`PATCH`); see the metadata guide for add, overwrite, and delete behavior.
| Constraint | Limit |
| ------------ | ---------------------------------- |
| Maximum keys | 50 |
| Key length | 40 characters |
| Key format | No square brackets (`[` or `]`) |
| Value length | 500 characters (stored as strings) |
**Type coercion:** Numbers and booleans are converted to strings (for example `42` → `"42"`, `true` → `"true"`). Nested objects and arrays are rejected.
For complete examples and PATCH semantics, see [Metadata](/api-reference/common/metadata).
## Important Notes
If you want structured JSON you need to pass the specific parser to the API before making the requests. For example if you want to get the JSON from Google Search you will pass this parser `"parser": {"id": "@olostep/google-search"}`
You can create your own parsers to retrieve the data you want from any page. Reach out to `info@olostep.com` to learn more.
## Conclusion
The batch endpoint is useful if you need to get data from many URLs in a short period of time. You need to already have the list of urls you want to get the data from.
Common applications can be:
* Price tracking services that monitor multiple e-commerce sites for price changes on products
* Website monitoring tools that check for content updates across numerous pages
* Data aggregation for concert organizers tracking ticket availability across multiple venues
* Search engines gathering and indexing content from many websites simultaneously
* News aggregators collecting articles from various publications
* Real estate platforms monitoring property listings across multiple sites
* Job boards aggregating openings from company career pages
* Financial data services tracking stock prices and market information
* Social media monitoring tools analyzing mentions across different platforms
* Academic research gathering data from multiple sources for analysis
It's better to use the batch endpoint if you want the content from 100 to 10k urls at a time. If you need to scrape less than 50 urls at a time we recommended using the scrape endpoint and submit the urls in parallel since it's faster than the batch endpoint.
## Pricing
Batch costs 1 credit per URL.
# Crawl
Source: https://docs.olostep.com/features/crawls
Crawl a URL and get the content from all subpages
Through the Olostep `/v1/crawls` endpoint you can crawl a website and get the content from all the pages.
* Crawl a website and get the content from all subpages (or limit the depth of the crawl)
* Use special patterns to crawl specific pages (e.g. `/blog/**`)
* Pass a `webhook_url` to get notified when the crawl is completed
* Search query to only find specific pages and sort by relevance
For API details see the [Crawl Endpoint API Reference](/api-reference/crawls/create).
## Installation
```python Python theme={null}
pip install olostep
```
```javascript Node theme={null}
npm install olostep
```
```bash cURL theme={null}
# curl is available by default on macOS, Linux, and Windows
```
```javascript Node (API) theme={null}
npm install node-fetch
```
```bash Python (API) theme={null}
pip install requests
```
## Start a crawl
Provide the starting URL, include/exclude URL globs, and `max_pages`. Optional: `max_depth`, `include_external`, `include_subdomain`, `search_query`, `top_n`, `webhook_url`, `timeout`.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
crawl = client.crawls.create(
start_url="https://olostep.com",
max_pages=100,
include_urls=["/**"],
exclude_urls=["/collections/**"],
include_external=False,
)
print(crawl.id, crawl.status)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const crawl = await client.crawls.create({
url: 'https://olostep.com',
maxPages: 100,
includeUrls: ['/**'],
excludeUrls: ['/collections/**'],
includeExternal: false,
})
console.log(crawl.id, crawl.status)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/crawls" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"start_url": "https://olostep.com",
"max_pages": 100,
"include_urls": ["/**"],
"exclude_urls": ["/collections/**"]
}'
```
```bash CLI theme={null}
olostep crawl "https://olostep.com" \
--max-pages 100 \
--include-url "/**" \
--exclude-url "/collections/**" \
--formats markdown,html
```
```js Node (API) theme={null}
const API_URL = 'https://api.olostep.com'
const res = await fetch(`${API_URL}/v1/crawls`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
start_url: 'https://olostep.com',
max_pages: 100,
include_urls: ['/**'],
exclude_urls: ['/collections/**']
})
})
console.log(await res.json())
```
```python Python (API) theme={null}
import time, json
API_URL = 'https://api.olostep.com'
API_KEY = ''
HEADERS = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {API_KEY}'
}
data = {
"start_url": "https://olostep.com",
"max_pages": 100,
"include_urls": ["/**"],
"exclude_urls": ["/collections/**"],
"include_external": False
}
res = requests.post(f"{API_URL}/v1/crawls", headers=HEADERS, json=data)
crawl = res.json()
print(json.dumps(crawl, indent=2))
```
Since everything in Olostep is an object, you will receive a `crawl` object in response. The `crawl` object has a few properties like `id` and `status`, which you can use to track the crawl.
## Check crawl status
Poll the crawl to track progress until `status` is `completed`.
```python Python theme={null}
# Using the crawl object from the previous step
info = crawl.info()
print(info.status, info.pages_count)
# Or wait until completed
crawl.wait_till_done(check_every_n_secs=5)
```
```js Node theme={null}
// Using the crawl object from the previous step
const info = await crawl.info()
console.log(info.status, info.pages_count)
// Or wait until completed
await crawl.waitTillDone({ checkEveryNSecs: 5 })
```
```bash cURL theme={null}
curl -s -X GET "https://api.olostep.com/v1/crawls/" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
```js Node (API) theme={null}
const crawlId = ''
const status = await fetch(`${API_URL}/v1/crawls/${crawlId}`, {
headers: { 'Authorization': 'Bearer ' }
}).then(r => r.json())
console.log(status)
```
```python Python (API) theme={null}
import time
def get_crawl_info(crawl_id):
return requests.get(f'{API_URL}/v1/crawls/{crawl_id}', headers=HEADERS).json()
crawl_id = crawl['id']
while True:
info = get_crawl_info(crawl_id)
print(info['status'], info.get('pages_count'))
if info['status'] == 'completed':
break
time.sleep(5)
```
Alternatively, you can pass a `webhook_url` when starting the crawl to be notified when the crawl is completed.
## List pages (paginate/stream with cursor)
Fetch pages and iterate using `cursor` and `limit`. Works while the crawl is `in_progress` or `completed`.
```python Python theme={null}
# Iterate all pages (auto-waits for crawl completion, handles pagination)
for page in crawl.pages():
print(page.url, page.retrieve_id)
```
```js Node theme={null}
// Iterate all pages (auto-waits for crawl completion, handles pagination)
for await (const page of crawl.pages()) {
console.log(page.url, page.retrieve_id)
}
```
```bash cURL theme={null}
curl -s -G "https://api.olostep.com/v1/crawls//pages" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
--data-urlencode "cursor=0" \
--data-urlencode "limit=10"
```
```js Node (API) theme={null}
let cursor = 0
while (true) {
const pages = await fetch(`${API_URL}/v1/crawls/${crawlId}/pages?cursor=${cursor}&limit=10`, {
headers: { 'Authorization': 'Bearer ' }
}).then(r => r.json())
pages.pages.forEach(p => console.log(p.url, p.retrieve_id))
if (pages.cursor === undefined) break
cursor = pages.cursor
}
```
```python Python (API) theme={null}
def get_pages(crawl_id, cursor=None, limit=10, search_query=None):
params = {
'cursor': cursor,
'limit': limit
}
return requests.get(f'{API_URL}/v1/crawls/{crawl_id}/pages', headers=HEADERS, params=params).json()
cursor = 0
while True:
page_batch = get_pages(crawl_id, cursor=cursor, limit=10)
for page in page_batch['pages']:
print(page['url'], page['retrieve_id'])
if 'cursor' not in page_batch:
break
cursor = page_batch['cursor']
time.sleep(5)
```
## Search query (limit to top N relevant)
Use `search_query` at start, and optionally filter listing with `search_query`. Limit per-page exploration with `top_n`.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
crawl = client.crawls.create(
start_url="https://olostep.com",
max_pages=100,
include_urls=["/**"],
search_query="contact us",
top_n=5,
)
for page in crawl.pages(search_query="contact us"):
print(page.url)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const crawl = await client.crawls.create({
url: 'https://olostep.com',
maxPages: 100,
includeUrls: ['/**'],
searchQuery: 'contact us',
topN: 5,
})
for await (const page of crawl.pages()) {
console.log(page.url)
}
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/crawls" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"start_url": "https://olostep.com",
"max_pages": 100,
"include_urls": ["/**"],
"search_query": "contact us",
"top_n": 5
}'
```
```bash CLI theme={null}
olostep crawl "https://olostep.com" \
--max-pages 100 \
--include-url "/**" \
--search-query "contact us" \
--top-n 5
```
```js Node (API) theme={null}
await fetch(`${API_URL}/v1/crawls`, { method: 'POST', headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' }, body: JSON.stringify({ start_url: 'https://olostep.com', max_pages: 100, include_urls: ['/**'], search_query: 'contact us', top_n: 5 }) })
```
```python Python (API) theme={null}
data = {
"start_url": "https://olostep.com",
"max_pages": 100,
"include_urls": ["/**"],
"search_query": "contact us",
"top_n": 5
}
crawl = requests.post(f'{API_URL}/v1/crawls', headers=HEADERS, json=data).json()
pages = requests.get(f"{API_URL}/v1/crawls/{crawl['id']}/pages", headers=HEADERS, params={'search_query': 'contact us'}).json()
print(len(pages['pages']))
```
## Retrieve content
Use each page's `retrieve_id` with `/v1/retrieve` to fetch `html_content` and/or `markdown_content`.
```python Python theme={null}
# Retrieve content for each crawled page
for page in crawl.pages():
content = page.retrieve(["markdown"])
print(content.markdown_content)
```
```js Node theme={null}
// Retrieve content for each crawled page
for await (const page of crawl.pages()) {
const content = await client.retrieve.get(page.retrieve_id, ['markdown'])
console.log(content.markdown_content)
}
```
```bash cURL theme={null}
curl -s -G "https://api.olostep.com/v1/retrieve" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
--data-urlencode "retrieve_id="
```
```bash CLI theme={null}
# `olostep crawl` retrieves content automatically. To fetch a single
# page by its retrieve_id later:
olostep scrape-get
```
```js Node (API) theme={null}
const retrieved = await fetch(`${API_URL}/v1/retrieve?retrieve_id=`, { headers: { 'Authorization': 'Bearer ' } }).then(r => r.json())
console.log(retrieved.markdown_content)
```
```python Python (API) theme={null}
def retrieve_content(retrieve_id):
return requests.get(f"{API_URL}/v1/retrieve", headers=HEADERS, params={"retrieve_id": retrieve_id}).json()
for page in get_pages(crawl['id'], limit=5)['pages']:
retrieved = retrieve_content(page['retrieve_id'])
print(retrieved.get('markdown_content'))
```
## Notes
* Pagination is cursor-based; repeat requests until `cursor` is absent.
* Content fields on `/v1/crawls/{crawl_id}/pages` are deprecated; prefer `/v1/retrieve`.
* Webhooks: set `webhook_url` to receive a POST when the crawl completes.
## Pricing
Crawl costs 1 credit per page crawled.
# Files
Source: https://docs.olostep.com/features/files
Upload and manage JSON files for use as context in API requests
Through the Olostep `/v1/files` endpoint you can upload JSON files that can be used as context in your API requests. This allows you to provide structured data to enhance your scrapes, answers, and other operations.
* Upload JSON files up to 200MB
* Files are automatically validated for proper JSON format
* Use files as context in scrapes, answers, and other endpoints
* Files expire after 30 days
* Secure pre-signed URL upload process
For API details see the [Files Endpoint API Reference](/api-reference/files/create).
## Installation
```python Python theme={null}
# pip install requests
import requests
```
```js Node theme={null}
// npm install node-fetch
// ESM
import fetch from 'node-fetch'
// CommonJS
const fetch = require('node-fetch')
```
```bash cURL theme={null}
# macOS: builtin curl is fine
```
## Upload a file
The file upload process consists of two steps:
1. **Create upload URL**: Request a pre-signed URL for uploading your file
2. **Complete upload**: Upload your file to the pre-signed URL, then call the complete endpoint to validate and finalize
### Step 1: Create upload URL
First, create an upload URL by providing the filename and optional purpose. The `purpose` parameter supports only two values: `"context"` (default) or `"batch"`.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
# Step 1: Create upload URL
payload = {
"filename": "my-data.json",
"purpose": "context" # Optional, defaults to "context". Supported values: "context" or "batch"
}
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
response = requests.post(f"{API_URL}/files", headers=headers, json=payload)
upload_data = response.json()
print(json.dumps(upload_data, indent=2))
# Response includes: id, upload_url, expires_in
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const res = await fetch(`${API_URL}/files`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
filename: 'my-data.json',
purpose: 'context' // Optional, defaults to "context". Supported values: "context" or "batch"
})
})
const uploadData = await res.json()
console.log(uploadData)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/files" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"filename": "my-data.json",
"purpose": "context"
}'
# Example with "batch" purpose:
curl -s -X POST "https://api.olostep.com/v1/files" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"filename": "batch-data.json",
"purpose": "batch"
}'
```
The response includes a pre-signed `upload_url` that expires in 10 minutes:
```json theme={null}
{
"id": "file_abc123xyz789",
"object": "file.upload",
"created": 1760329882,
"upload_url": "https://olostep-files.s3.amazonaws.com/files/...",
"expires_in": 600
}
```
### Step 2: Upload file and complete
Upload your JSON file to the pre-signed URL, then call the complete endpoint to validate and finalize the upload.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
# After getting upload_url from Step 1
file_id = upload_data["id"]
upload_url = upload_data["upload_url"]
# Prepare your JSON data
json_data = {
"users": [
{"name": "John Doe", "email": "john@example.com"},
{"name": "Jane Smith", "email": "jane@example.com"}
]
}
# Step 2a: Upload file to pre-signed URL
upload_response = requests.put(
upload_url,
data=json.dumps(json_data),
headers={"Content-Type": "application/json"}
)
upload_response.raise_for_status()
# Step 2b: Complete the upload
complete_response = requests.post(
f"{API_URL}/files/{file_id}/complete",
headers={"Authorization": f"Bearer {API_KEY}"}
)
file_info = complete_response.json()
print(json.dumps(file_info, indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
// After getting upload_url from Step 1
const fileId = uploadData.id
const uploadUrl = uploadData.upload_url
// Prepare your JSON data
const jsonData = {
users: [
{ name: 'John Doe', email: 'john@example.com' },
{ name: 'Jane Smith', email: 'jane@example.com' }
]
}
// Step 2a: Upload file to pre-signed URL
await fetch(uploadUrl, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(jsonData)
})
// Step 2b: Complete the upload
const completeRes = await fetch(`${API_URL}/files/${fileId}/complete`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ' }
})
const fileInfo = await completeRes.json()
console.log(fileInfo)
```
```bash cURL theme={null}
# Step 2a: Upload file to pre-signed URL
curl -X PUT "$UPLOAD_URL" \
-H "Content-Type: application/json" \
-d @my-data.json
# Step 2b: Complete the upload
curl -s -X POST "https://api.olostep.com/v1/files/$FILE_ID/complete" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
The complete endpoint validates the JSON file and returns file metadata:
```json theme={null}
{
"id": "file_abc123xyz789",
"object": "file",
"created": 1760329882,
"filename": "my-data.json",
"bytes": 1024,
"purpose": "context",
"status": "completed"
}
```
## Retrieve file metadata by ID
Retrieve metadata for a file by its ID.
```python Python theme={null}
file_id = "file_abc123xyz789"
response = requests.get(
f"{API_URL}/files/{file_id}",
headers={"Authorization": f"Bearer {API_KEY}"}
)
file_info = response.json()
print(json.dumps(file_info, indent=2))
```
```js Node theme={null}
const fileId = 'file_abc123xyz789'
const res = await fetch(`${API_URL}/files/${fileId}`, {
headers: { 'Authorization': 'Bearer ' }
})
const fileInfo = await res.json()
console.log(fileInfo)
```
```bash cURL theme={null}
curl -s -X GET "https://api.olostep.com/v1/files/$FILE_ID" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
## Retrieve file object by ID
Get a pre-signed URL to download the JSON content of a completed file. Optionally specify the expiration time for the download URL using the `expires_in` query parameter (defaults to 600 seconds / 10 minutes).
```python Python theme={null}
file_id = "file_abc123xyz789"
# Get download URL (default expiration: 600 seconds)
response = requests.get(
f"{API_URL}/files/{file_id}/content",
headers={"Authorization": f"Bearer {API_KEY}"}
)
download_info = response.json()
download_url = download_info["download_url"]
# Download the file content using the pre-signed URL
file_response = requests.get(download_url)
file_content = file_response.json()
print(json.dumps(file_content, indent=2))
# Example with custom expiration (3600 seconds = 1 hour)
response = requests.get(
f"{API_URL}/files/{file_id}/content?expires_in=3600",
headers={"Authorization": f"Bearer {API_KEY}"}
)
download_info = response.json()
print(f"Download URL expires in: {download_info['expires_in']} seconds")
```
```js Node theme={null}
const fileId = 'file_abc123xyz789'
// Get download URL (default expiration: 600 seconds)
const res = await fetch(`${API_URL}/files/${fileId}/content`, {
headers: { 'Authorization': 'Bearer ' }
})
const downloadInfo = await res.json()
const downloadUrl = downloadInfo.download_url
// Download the file content using the pre-signed URL
const fileRes = await fetch(downloadUrl)
const fileContent = await fileRes.json()
console.log(fileContent)
// Example with custom expiration (3600 seconds = 1 hour)
const customRes = await fetch(`${API_URL}/files/${fileId}/content?expires_in=3600`, {
headers: { 'Authorization': 'Bearer ' }
})
const customDownloadInfo = await customRes.json()
console.log(`Download URL expires in: ${customDownloadInfo.expires_in} seconds`)
```
```bash cURL theme={null}
# Get download URL (default expiration: 600 seconds)
curl -s -X GET "https://api.olostep.com/v1/files/$FILE_ID/content" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
# Get download URL with custom expiration (3600 seconds = 1 hour)
curl -s -X GET "https://api.olostep.com/v1/files/$FILE_ID/content?expires_in=3600" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
# Download the file using the pre-signed URL
curl -s "$DOWNLOAD_URL"
```
The response includes a pre-signed `download_url` that expires after the specified time:
```json theme={null}
{
"id": "file_abc123xyz789",
"object": "file",
"created": 1760329882,
"filename": "my-data.json",
"bytes": 1024,
"download_url": "https://olostep-files.s3.amazonaws.com/files/...",
"expires_in": 600
}
```
## List files
List all completed files for your team. Optionally filter by purpose (supported values: `"context"` or `"batch"`).
```python Python theme={null}
# List all files
response = requests.get(
f"{API_URL}/files",
headers={"Authorization": f"Bearer {API_KEY}"}
)
files = response.json()
print(json.dumps(files, indent=2))
# List files filtered by purpose
response = requests.get(
f"{API_URL}/files?purpose=context",
headers={"Authorization": f"Bearer {API_KEY}"}
)
context_files = response.json()
print(json.dumps(context_files, indent=2))
```
```js Node theme={null}
// List all files
const res = await fetch(`${API_URL}/files`, {
headers: { 'Authorization': 'Bearer ' }
})
const files = await res.json()
console.log(files)
// List files filtered by purpose
const contextRes = await fetch(`${API_URL}/files?purpose=context`, {
headers: { 'Authorization': 'Bearer ' }
})
const contextFiles = await contextRes.json()
console.log(contextFiles)
```
```bash cURL theme={null}
# List all files
curl -s -X GET "https://api.olostep.com/v1/files" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
# List files filtered by purpose
curl -s -X GET "https://api.olostep.com/v1/files?purpose=context" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
The response includes a list of files:
```json theme={null}
{
"object": "list",
"data": [
{
"id": "file_abc123xyz789",
"object": "file",
"created": 1760329882,
"filename": "my-data.json",
"bytes": 1024,
"purpose": "context",
"status": "completed"
}
]
}
```
## Delete a file
Delete a file and its associated data from storage.
```python Python theme={null}
file_id = "file_abc123xyz789"
response = requests.delete(
f"{API_URL}/files/{file_id}",
headers={"Authorization": f"Bearer {API_KEY}"}
)
result = response.json()
print(json.dumps(result, indent=2))
```
```js Node theme={null}
const fileId = 'file_abc123xyz789'
const res = await fetch(`${API_URL}/files/${fileId}`, {
method: 'DELETE',
headers: { 'Authorization': 'Bearer ' }
})
const result = await res.json()
console.log(result)
```
```bash cURL theme={null}
curl -s -X DELETE "https://api.olostep.com/v1/files/$FILE_ID" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
## Complete upload example (context purpose)
Here's a complete example that uploads a JSON file with `purpose="context"`:
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
# Step 1: Create upload URL
create_response = requests.post(
f"{API_URL}/files",
headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
json={"filename": "user-data.json", "purpose": "context"}
)
upload_data = create_response.json()
file_id = upload_data["id"]
upload_url = upload_data["upload_url"]
# Step 2: Prepare and upload JSON data
json_data = {
"users": [
{"id": 1, "name": "Alice", "role": "admin"},
{"id": 2, "name": "Bob", "role": "user"}
]
}
upload_response = requests.put(
upload_url,
data=json.dumps(json_data),
headers={"Content-Type": "application/json"}
)
upload_response.raise_for_status()
# Step 3: Complete the upload
complete_response = requests.post(
f"{API_URL}/files/{file_id}/complete",
headers={"Authorization": f"Bearer {API_KEY}"}
)
file_info = complete_response.json()
print(f"File uploaded successfully: {file_info['id']}")
print(f"File size: {file_info['bytes']} bytes")
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
// Step 1: Create upload URL
const createRes = await fetch(`${API_URL}/files`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({ filename: 'user-data.json', purpose: 'context' })
})
const uploadData = await createRes.json()
const fileId = uploadData.id
const uploadUrl = uploadData.upload_url
// Step 2: Prepare and upload JSON data
const jsonData = {
users: [
{ id: 1, name: 'Alice', role: 'admin' },
{ id: 2, name: 'Bob', role: 'user' }
]
}
await fetch(uploadUrl, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(jsonData)
})
// Step 3: Complete the upload
const completeRes = await fetch(`${API_URL}/files/${fileId}/complete`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ' }
})
const fileInfo = await completeRes.json()
console.log(`File uploaded successfully: ${fileInfo.id}`)
console.log(`File size: ${fileInfo.bytes} bytes`)
```
## Upload batch file example
Here's an example that uploads a JSON file with `purpose="batch"` containing valid batch data that can be used with the `/v1/batches` endpoint:
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
# Step 1: Create upload URL with purpose="batch"
create_response = requests.post(
f"{API_URL}/files",
headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
json={"filename": "batch-items.json", "purpose": "batch"}
)
upload_data = create_response.json()
file_id = upload_data["id"]
upload_url = upload_data["upload_url"]
# Step 2: Prepare batch JSON data (valid format for /v1/batches endpoint)
batch_data = {
"items": [
{"custom_id": "item-1", "url": "https://www.google.com/search?q=stripe&gl=us&hl=en"},
{"custom_id": "item-2", "url": "https://www.google.com/search?q=paddle&gl=us&hl=en"},
{"custom_id": "item-3", "url": "https://www.google.com/search?q=payment+gateway&gl=us&hl=en"}
],
"parser": {"id": "@olostep/google-search"},
"country": "US"
}
upload_response = requests.put(
upload_url,
data=json.dumps(batch_data),
headers={"Content-Type": "application/json"}
)
upload_response.raise_for_status()
# Step 3: Complete the upload
complete_response = requests.post(
f"{API_URL}/files/{file_id}/complete",
headers={"Authorization": f"Bearer {API_KEY}"}
)
file_info = complete_response.json()
print(f"Batch file uploaded successfully: {file_info['id']}")
print(f"File size: {file_info['bytes']} bytes")
print(f"Purpose: {file_info['purpose']}")
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
// Step 1: Create upload URL with purpose="batch"
const createRes = await fetch(`${API_URL}/files`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({ filename: 'batch-items.json', purpose: 'batch' })
})
const uploadData = await createRes.json()
const fileId = uploadData.id
const uploadUrl = uploadData.upload_url
// Step 2: Prepare batch JSON data (valid format for /v1/batches endpoint)
const batchData = {
items: [
{ custom_id: 'item-1', url: 'https://www.google.com/search?q=stripe&gl=us&hl=en' },
{ custom_id: 'item-2', url: 'https://www.google.com/search?q=paddle&gl=us&hl=en' },
{ custom_id: 'item-3', url: 'https://www.google.com/search?q=payment+gateway&gl=us&hl=en' }
],
parser: { id: '@olostep/google-search' },
country: 'US'
}
await fetch(uploadUrl, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(batchData)
})
// Step 3: Complete the upload
const completeRes = await fetch(`${API_URL}/files/${fileId}/complete`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ' }
})
const fileInfo = await completeRes.json()
console.log(`Batch file uploaded successfully: ${fileInfo.id}`)
console.log(`File size: ${fileInfo.bytes} bytes`)
console.log(`Purpose: ${fileInfo.purpose}`)
```
```bash cURL theme={null}
# Step 1: Create upload URL with purpose="batch"
curl -s -X POST "https://api.olostep.com/v1/files" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"filename": "batch-items.json",
"purpose": "batch"
}'
# Step 2: Upload batch JSON data (save to batch-items.json first)
# batch-items.json content:
# {
# "items": [
# {"custom_id": "item-1", "url": "https://www.google.com/search?q=stripe&gl=us&hl=en"},
# {"custom_id": "item-2", "url": "https://www.google.com/search?q=paddle&gl=us&hl=en"},
# {"custom_id": "item-3", "url": "https://www.google.com/search?q=payment+gateway&gl=us&hl=en"}
# ],
# "parser": {"id": "@olostep/google-search"},
# "country": "US"
# }
curl -X PUT "$UPLOAD_URL" \
-H "Content-Type: application/json" \
-d @batch-items.json
# Step 3: Complete the upload
curl -s -X POST "https://api.olostep.com/v1/files/$FILE_ID/complete" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
The uploaded batch file contains a valid JSON structure that matches the `/v1/batches` endpoint format:
* `items`: Array of objects with `custom_id` and `url` fields
* `parser`: Optional parser configuration
* `country`: Optional country code
This file can be used as input for batch processing operations.
## File requirements
* **File format**: Only JSON files are supported (`.json` extension required)
* **File size**: Maximum 200MB per file
* **Expiration**: Files expire after 30 days
* **Upload URL**: Pre-signed URLs expire after 10 minutes
* **Purpose parameter**: Only supports `"context"` or `"batch"` values (defaults to `"context"`)
## Pricing
File uploads are free. Files are stored securely and automatically expire after 30 days.
# Maps
Source: https://docs.olostep.com/features/maps
Get all the URLs on a website.
Through the Olostep `/v1/maps` endpoint you can get all the URLs on a website. This is useful for content discovery, site structure analysis (e.g., SEO), or deciding which URLs you want to scrape next.
* Get all URLs on a website (including sitemaps and discovered links)
* Use special patterns to include/exclude paths (e.g. `/blog/**`)
* Paginate large responses with `cursor` (up to 10MB per response)
* Limit volume with `top_n`
For API details see the [Map Endpoint API Reference](/api-reference/maps/create).
## Installation
```python Python theme={null}
pip install olostep
```
```javascript Node theme={null}
npm install olostep
```
```bash cURL theme={null}
# curl is available by default on macOS, Linux, and Windows
```
```javascript Node (API) theme={null}
npm install node-fetch
```
```bash Python (API) theme={null}
pip install requests
```
## Usage
Send a POST request with the website `url`. Optionally pass `include_urls`, `exclude_urls` (glob patterns), and `top_n`.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
sitemap = client.maps.create(url="https://docs.olostep.com")
for url in sitemap.urls():
print(url)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const map = await client.maps.create({ url: 'https://docs.olostep.com' })
for await (const url of map.urls()) {
console.log(url)
}
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/maps" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.olostep.com"
}'
```
```bash CLI theme={null}
olostep map "https://docs.olostep.com"
```
```js Node (API) theme={null}
const res = await fetch('https://api.olostep.com/v1/maps', {
method: 'POST',
headers: {
'Authorization': 'Bearer ',
'Content-Type': 'application/json'
},
body: JSON.stringify({ url: 'https://docs.olostep.com' })
})
console.log(await res.json())
```
```python Python (API) theme={null}
import requests
import json
endpoint = "https://api.olostep.com/v1/maps"
payload = {
"url": "https://docs.olostep.com"
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
response = requests.post(endpoint, json=payload, headers=headers)
print(json.dumps(response.json(), indent=2))
```
The response time is typically within seconds but can take up to 120 seconds for more complex websites. It can extract all URLs from a website, even backlinks and those not present in the Sitemaps. You can also also decide the URLs paths you want to include or exclude from the response.
By default the endpoint returns around 100k URLs in a single call (10MB max). If the response includes more data, the API returns a `cursor` parameter which can be used for pagination and getting the subsequent URLs. For more details refer to the [API Reference](/api-reference/maps/create#body-cursor)
This endpoint is particularly useful when you need to:
* Discover all content pages on a website
* Analyze site structure and hierarchy
* Prepare URLs for batch processing
* Decide which specific URLs to scrape
For more fine-grained control over the URLs returned you can use the params `include_urls` and `exclude_urls`.
### Example
Let's say that from [www.brex.com](http://www.brex.com) you want to extract all the urls that have the paths after `/product/` e.g `https://www.brex.com/product/api/no-code` but also include `www.brex.com/product`.
You can use the following code:
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
sitemap = client.maps.create(
url="https://www.brex.com/",
include_urls=["/product", "/product/**"],
top_n=100000,
)
for url in sitemap.urls():
print(url)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const map = await client.maps.create({
url: 'https://www.brex.com/',
includeUrls: ['/product', '/product/**'],
topN: 100000,
})
for await (const url of map.urls()) {
console.log(url)
}
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/maps" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.brex.com/",
"include_urls": ["/product", "/product/**"],
"top_n": 100000
}'
```
```bash CLI theme={null}
olostep map "https://www.brex.com/" \
--include-url "/product" \
--include-url "/product/**" \
--top-n 100000
```
```js Node (API) theme={null}
const endpoint = 'https://api.olostep.com/v1/maps'
const res = await fetch(endpoint, {
method: 'POST',
headers: {
'Authorization': 'Bearer ',
'Content-Type': 'application/json'
},
body: JSON.stringify({
url: 'https://www.brex.com/',
include_urls: ['/product', '/product/**'],
top_n: 100000
})
})
console.log(await res.text())
```
```python Python (API) theme={null}
import requests
endpoint = "https://api.olostep.com/v1/maps"
payload = {
"url": "https://www.brex.com/",
"include_urls": ["/product", "/product/**"],
"top_n": 100000
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
response = requests.post(endpoint, json=payload, headers=headers)
print(response.text)
```
## Conclusion
The maps endpoint is a powerful tool for content discovery and site analysis. It provides a comprehensive list of URLs on a website, enabling you to extract content from specific pages or analyze the site structure. This endpoint is particularly useful for SEO professionals, content marketers, AI agents who need to analyze website content or structure.
## Pricing
Map costs 1 credit. Then for every extra 1000 URLs returned in the response, an additional credit is billed.
# Monitors
Source: https://docs.olostep.com/features/monitors
Monitor pages on a schedule and send change alerts
Through the Olostep `/v1/monitors` endpoint you can create persistent monitors that run on a fixed schedule, detect page changes, and notify you through email, Slack, SMS, or a dedicated webhook.
* Create a monitor from a natural language `query`
* Scope sources with `source_policy`
* Run checks on natural-language schedules (minimum every 10 minutes, UTC)
* Configure `notification.channels` and optional `webhook` delivery
* Stream provisioning progress with Server-Sent Events (`?stream=1`)
* List, inspect, update, pause, resume, and delete monitors
* Read snapshot events, planning artifacts, run logs, and live agent logs
By default, every monitor run captures a **full snapshot** of the monitored page — a complete picture of its current state at that moment. If you want the monitor to surface only what's new or changed between runs (deltas) instead of the full state, express that intent in the `query`.
## Installation
```python Python theme={null}
# pip install requests
import requests
```
```js Node theme={null}
// npm install node-fetch
// ESM
import fetch from 'node-fetch'
// CommonJS
const fetch = require('node-fetch')
```
```bash cURL theme={null}
# macOS: builtin curl is fine
```
## Create a monitor
Create a monitor with `POST /v1/monitors`. The API validates your input, reserves a monitor record, provisions a shadow agent, generates a workflow spec, queues DAG planning, and creates a recurring schedule.
* `query` is required — describe what to watch in natural language.
* `frequency` is optional and defaults to `every hour`. Use scheduling phrases such as `every day at 9am` (schedules run in **UTC**; minimum interval is **10 minutes**).
* `source_policy` optionally constrains `include_urls`, `exclude_urls`, `include_domains`, and `exclude_domains`.
* `notification` configures when and how to alert (`events` + `channels`). Channel delivery is resolved at runtime by the monitor pipeline — you do not pass recipients into the DAG.
* `webhook` is a separate object (`{ "url": "https://…" }`) for HTTP callbacks in addition to `notification.channels`.
* `output_schema` optionally enforces structured extraction (valid JSON Schema).
The create response is HTTP `202` with `status: provisioning`. The monitor moves to `active` after planning resolves `tracked` targets. Poll `GET /v1/monitors/:monitor_id` or pass `?stream=1` (or `Accept: text/event-stream`) to follow provisioning phases and spec reasoning tokens over SSE.
### Example request
You only need `query` and `frequency`. Notification channels and webhooks can be added later with `POST /v1/monitors/:monitor_id`.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
payload = {
"query": "Notify me when a new startup launches on Y Combinator Launches",
"frequency": "every 20 minutes",
}
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
response = requests.post(f"{API_URL}/monitors", headers=headers, json=payload)
print(response.status_code)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const res = await fetch(`${API_URL}/monitors`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
query: 'Notify me when a new startup launches on Y Combinator Launches',
frequency: 'every 20 minutes',
}),
})
console.log(res.status)
console.log(await res.json())
```
```bash cURL theme={null}
curl -sS -X POST "https://api.olostep.com/v1/monitors" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "Notify me when a new startup launches on Y Combinator Launches",
"frequency": "every 20 minutes"
}'
```
### Response
Successful creation (non-streaming) returns HTTP `202` with a monitor object. `tracked` is empty until planning finishes; poll `GET /v1/monitors/:monitor_id` until `status` is `active` and `tracked.urls` is populated.
```json theme={null}
{
"id": "monitor_biglavgvq3",
"object": "monitor",
"query": "Notify me when a new startup launches on Y Combinator Launches",
"tracked": {
"type": null,
"urls": [],
"web_query": null
},
"source_policy": {},
"schedule": {
"frequency": "every 20 minutes",
"cron": "7/20 * * * ? *",
"timezone": "UTC",
"next_run_at": null
},
"notification": {
"events": [],
"channels": []
},
"webhook": null,
"output_schema": {},
"status": "provisioning",
"error_message": null,
"last_run": null,
"agent": {
"id": "agent_forward_deployed_0_fda_nlkxhr5kto"
},
"metadata": {},
"created": 1780063068,
"updated": 1780063071
}
```
### Structured monitor output
Set `output_schema` when you want extraction results to follow a specific JSON structure. The schema must be valid JSON Schema.
### Provisioning stream
Add `?stream=1` or send `Accept: text/event-stream` to receive SSE events while the monitor is created:
| Event | Description |
| ----------------- | ------------------------------------------------------- |
| `phase` | Provisioning step (`running`, `done`, or `failed`) |
| `reasoning_token` | Incremental spec-design text |
| `reasoning_reset` | Truncates buffered reasoning after a failed LLM attempt |
| `complete` | Final monitor object (same shape as the `202` response) |
| `error` | Terminal failure |
## Notifications and webhooks
Alerts are configured on the monitor record and resolved at runtime — do not embed channel targets in the monitoring `query`.
### `notification`
| Field | Description |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `events` | Which run outcomes should trigger delivery. See [notification events](#notification-events) below. If you set `channels` and omit `events`, defaults to both `changed` and `first_snapshot`. |
| `channels` | List of `{ "type", "target", "events"? }` objects |
#### Notification events
Use `events` to state **when** you want to be notified. Allowed values:
| Event | Meaning |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `changed` | Notify when the monitor detects a **change** compared to the previous snapshot (for example new content, updated price, or a diff the pipeline classifies as changed). |
| `first_snapshot` | Notify when the monitor takes its **first snapshot** — the initial baseline run that stores current content before later comparisons. |
You can include one or both. For example, `["changed"]` alerts only on updates after the baseline; `["first_snapshot"]` confirms setup without waiting for a diff; `["changed", "first_snapshot"]` covers both.
Per-channel `events` on a channel object uses the same values and overrides the top-level list for that channel only.
Supported channel types:
| `type` | `target` format |
| ------- | ----------------------------------------------- |
| `email` | Valid email address |
| `slack` | Slack incoming webhook URL |
| `sms` | E.164 phone number (for example `+14155552671`) |
### `webhook`
Separate from `notification.channels`, `webhook.url` receives HTTP POST payloads when the monitor fires your callback URL. You can use both a webhook and channel notifications on the same monitor.
### Examples
Email only:
```json theme={null}
{
"query": "Watch for changes on https://example.com/terms",
"frequency": "every day at 10am",
"notification": {
"events": ["changed"],
"channels": [
{ "type": "email", "target": "legal@example.com" }
]
}
}
```
Webhook callback:
```json theme={null}
{
"query": "Watch for changes on https://example.com/terms",
"frequency": "every day at 10am",
"webhook": {
"url": "https://hooks.example.com/olostep-monitor"
}
}
```
SMS:
```json theme={null}
{
"query": "Alert me when https://status.example.com shows an incident",
"frequency": "every hour",
"notification": {
"channels": [
{ "type": "sms", "target": "+14155552671" }
]
}
}
```
## Source policy
Use `source_policy` to constrain which URLs and domains the planner may use.
```json theme={null}
{
"source_policy": {
"include_urls": ["https://example.com/pricing"],
"exclude_domains": ["ads.example.com"]
}
}
```
## Frequencies
Set `frequency` in natural language, for example:
* `every hour` (default when omitted)
* `every day at 9am`
* `every weekday at 14:30`
Rules:
* Must read as scheduling language (not an arbitrary monitor question).
* Minimum interval: **every 10 minutes**.
* Schedules are stored and executed in **UTC** (`schedule.timezone` is `UTC`).
* Maximum length: 50 characters.
The API derives a cron expression from your `frequency` text and exposes it on `schedule.cron`. When the monitor is `active`, `schedule.next_run_at` shows the next run in ISO 8601.
## List monitors
Retrieve all monitors for your team with `GET /v1/monitors`.
By default, deleted monitors are filtered out. Use `?include_deleted=true` to include them.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
headers = { "Authorization": f"Bearer {API_KEY}" }
response = requests.get(f"{API_URL}/monitors", headers=headers)
result = response.json()
print(f"Total monitors: {result['count']}")
print(json.dumps(result, indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const res = await fetch(`${API_URL}/monitors`, {
headers: { 'Authorization': 'Bearer ' }
})
const result = await res.json()
console.log(`Total monitors: ${result.count}`)
console.log(result.monitors)
```
```bash cURL theme={null}
curl -s -X GET "https://api.olostep.com/v1/monitors" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
### Response shape
```json theme={null}
{
"monitors": [
{
"id": "monitor_0wj35czpn7",
"object": "monitor",
"query": "Watch AirOps blog for new blog posts",
"tracked": {
"type": "urls",
"urls": ["https://www.airops.com/blog"],
"web_query": null
},
"source_policy": {},
"schedule": {
"frequency": "every hour",
"cron": "2 * * * ? *",
"timezone": "UTC",
"next_run_at": null
},
"notification": {
"channels": [],
"events": []
},
"webhook": null,
"output_schema": {},
"status": "paused",
"error_message": null,
"last_run": null,
"agent": { "id": "agent_forward_deployed_0_fda_x4822l9h3i" },
"metadata": {},
"created": 1780062756,
"updated": 1780063025
},
{
"id": "monitor_biglavgvq3",
"object": "monitor",
"query": "Notify me when a new startup launches on Y Combinator Launches",
"tracked": {
"type": "urls",
"urls": ["https://www.ycombinator.com/launches/"],
"web_query": null
},
"source_policy": {},
"schedule": {
"frequency": "every 20 minutes",
"cron": "7/20 * * * ? *",
"timezone": "UTC",
"next_run_at": "2026-05-29T14:27:00.000Z"
},
"notification": {
"channels": [],
"events": []
},
"webhook": null,
"output_schema": {},
"status": "active",
"error_message": null,
"last_run": null,
"agent": { "id": "agent_forward_deployed_0_fda_nlkxhr5kto" },
"metadata": {},
"created": 1780063068,
"updated": 1780063141
}
],
"count": 5
}
```
## Get a monitor
Retrieve a single monitor with `GET /v1/monitors/:monitor_id`.
The response includes `last_run` (latest snapshot summary) and `total_count` (snapshot count) unless you pass `include_total_count=false`. Add `include-diagram=true` to include a `mermaid_diagram` of the monitor DAG.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
MONITOR_ID = "monitor_biglavgvq3"
headers = { "Authorization": f"Bearer {API_KEY}" }
response = requests.get(f"{API_URL}/monitors/{MONITOR_ID}", headers=headers)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const monitorId = 'monitor_biglavgvq3'
const res = await fetch(`${API_URL}/monitors/${monitorId}`, {
headers: { 'Authorization': 'Bearer ' }
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X GET "https://api.olostep.com/v1/monitors/monitor_biglavgvq3" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
### Response shape
```json theme={null}
{
"id": "monitor_biglavgvq3",
"object": "monitor",
"query": "Notify me when a new startup launches on Y Combinator Launches",
"tracked": {
"type": "urls",
"urls": ["https://www.ycombinator.com/launches/"],
"web_query": null
},
"source_policy": {},
"schedule": {
"frequency": "every 20 minutes",
"cron": "7/20 * * * ? *",
"timezone": "UTC",
"next_run_at": "2026-05-29T14:27:00.000Z"
},
"notification": {
"channels": [],
"events": []
},
"webhook": null,
"output_schema": {},
"status": "active",
"error_message": null,
"last_run": {
"id": "run_iwsoafcpyx",
"status": "completed",
"change_detected": false,
"ran_at": "2026-05-29T14:03:15.963Z"
},
"agent": {
"id": "agent_forward_deployed_0_fda_nlkxhr5kto"
},
"metadata": {},
"created": 1780063068,
"updated": 1780063141,
"total_count": 1
}
```
## List monitor events
Use `GET /v1/monitors/:monitor_id/events` to list snapshot events for a monitor.
Pagination:
* `limit` (default `25`, max `100`)
* `cursor` (opaque token from `next_cursor`)
* `count_only=true` returns only `{ "total_count": N }`
Events are returned newest-first. Each item includes a short-lived pre-signed `snapshot_url`.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
MONITOR_ID = "monitor_biglavgvq3"
headers = { "Authorization": f"Bearer {API_KEY}" }
response = requests.get(
f"{API_URL}/monitors/{MONITOR_ID}/events?limit=10",
headers=headers,
)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const monitorId = 'monitor_biglavgvq3'
const res = await fetch(`${API_URL}/monitors/${monitorId}/events?limit=10`, {
headers: { 'Authorization': 'Bearer ' }
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X GET "https://api.olostep.com/v1/monitors/monitor_biglavgvq3/events?limit=10" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
### Response shape
```json theme={null}
{
"data": [
{
"id": "run_iwsoafcpyx",
"run_id": "run_iwsoafcpyx",
"created": 1780063395,
"changed": false,
"summary": "First snapshot of this monitor has been taken. Stored current content as baseline.",
"snapshot_url": "https://olostep-monitor-snapshots.s3.amazonaws.com/monitor_biglavgvq3/run_iwsoafcpyx_snapshot.json?X-Amz-Expires=600&..."
}
],
"has_more": false,
"next_cursor": null,
"total_count": 1
}
```
## Get monitor planning
Use `GET /v1/monitors/:monitor_id/planning` to inspect the FDA workflow spec and planner DAG after provisioning.
```json theme={null}
{
"spec": {
"saved_at": "2026-05-29T12:00:00.000000+00:00",
"status": "complete",
"goal": "Track pricing on example.com",
"reasoning": "...",
"constraints": "...",
"assumptions": "...",
"input": { "query": "...", "urls": ["https://www.ycombinator.com/launches/"] },
"output": { "type": "free_text" },
"chat_history": []
},
"dag": {
"user_query": "...",
"graph": { "nodes": [], "edges": [] },
"has_unresolved": false,
"unresolved": [],
"validation": { "is_valid": true, "attempts": 1, "history": [] }
}
}
```
## Get a monitor run
Use `GET /v1/monitors/:monitor_id/runs/:run_id` for snapshot metadata and parsed agent log events for one execution (`run_id` must start with `run_`).
```json theme={null}
{
"monitor_id": "monitor_biglavgvq3",
"run_id": "run_v7k2p9m3",
"snapshot": { "changed": true, "summary": "..." },
"log_group": "/aws/ecs/olostep-agents/...",
"events": [
{
"id": "...",
"ts": 1777960800123,
"message": "Run run_v7k2p9m3 completed. Files uploaded: 2",
"event": { "type": "run_complete", "run_id": "run_v7k2p9m3", "files_uploaded": 2 }
}
]
}
```
## Stream agent logs
Use `GET /v1/monitors/:monitor_id/agent-logs?stream=1` (or `Accept: text/event-stream`) to tail CloudWatch logs for the monitor's agent, filtered to this `monitor_id`.
Optional query parameter `since` is a millisecond timestamp (default: 30 minutes ago).
SSE event types: `ready`, `log`, `heartbeat`, `error`.
## Update a monitor
Update a monitor with `POST /v1/monitors/:monitor_id`.
Supported fields (include only what you want to change):
* `metadata` — merged with existing keys; empty string values delete keys
* `frequency` — recreates the internal schedule and sets `status` back to `active`
* `notification` — replaces the entire notification object
* `webhook` — pass `null` to remove
Returns `409` while `status` is `provisioning`.
When you add `notification.channels` without `events`, the API defaults `events` to `["changed", "first_snapshot"]`.
### Add email notification
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
MONITOR_ID = "monitor_biglavgvq3"
payload = {
"notification": {
"channels": [
{"type": "email", "target": "you@example.com"}
]
}
}
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
response = requests.post(
f"{API_URL}/monitors/{MONITOR_ID}",
headers=headers,
json=payload,
)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const monitorId = 'monitor_biglavgvq3'
const res = await fetch(`${API_URL}/monitors/${monitorId}`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
notification: {
channels: [{ type: 'email', target: 'you@example.com' }]
}
})
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/monitors/monitor_biglavgvq3" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"notification": {
"channels": [
{ "type": "email", "target": "you@example.com" }
]
}
}'
```
```json theme={null}
{
"id": "monitor_biglavgvq3",
"object": "monitor",
"query": "Notify me when a new startup launches on Y Combinator Launches",
"tracked": {
"type": "urls",
"urls": ["https://www.ycombinator.com/launches/"],
"web_query": null
},
"source_policy": {},
"schedule": {
"frequency": "every 20 minutes",
"cron": "7/20 * * * ? *",
"timezone": "UTC",
"next_run_at": "2026-05-29T14:27:00.000Z"
},
"notification": {
"events": ["changed", "first_snapshot"],
"channels": [
{ "type": "email", "target": "you@example.com" }
]
},
"webhook": null,
"output_schema": {},
"status": "active",
"error_message": null,
"last_run": null,
"agent": { "id": "agent_forward_deployed_0_fda_nlkxhr5kto" },
"metadata": {},
"created": 1780063068,
"updated": 1780064634
}
```
### Add webhook
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
MONITOR_ID = "monitor_biglavgvq3"
payload = {
"webhook": { "url": "https://webhook.site/your-unique-id" }
}
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
response = requests.post(
f"{API_URL}/monitors/{MONITOR_ID}",
headers=headers,
json=payload,
)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const monitorId = 'monitor_biglavgvq3'
const res = await fetch(`${API_URL}/monitors/${monitorId}`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
webhook: { url: 'https://webhook.site/your-unique-id' }
})
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/monitors/monitor_biglavgvq3" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"webhook": { "url": "https://webhook.site/your-unique-id" }
}'
```
```json theme={null}
{
"id": "monitor_biglavgvq3",
"object": "monitor",
"query": "Notify me when a new startup launches on Y Combinator Launches",
"tracked": {
"type": "urls",
"urls": ["https://www.ycombinator.com/launches/"],
"web_query": null
},
"source_policy": {},
"schedule": {
"frequency": "every 20 minutes",
"cron": "7/20 * * * ? *",
"timezone": "UTC",
"next_run_at": "2026-05-29T14:47:00.000Z"
},
"notification": {
"channels": [
{ "type": "email", "target": "you@example.com" }
],
"events": ["changed", "first_snapshot"]
},
"webhook": {
"url": "https://webhook.site/your-unique-id"
},
"output_schema": {},
"status": "active",
"error_message": null,
"last_run": null,
"agent": { "id": "agent_forward_deployed_0_fda_nlkxhr5kto" },
"metadata": {},
"created": 1780063068,
"updated": 1780065538
}
```
## Pause a monitor
Pause a monitor with `POST /v1/monitors/:monitor_id/pause`.
Pausing disables the underlying schedule and sets `status` to `paused`. Only monitors with `status: active` can be paused. The request body is empty.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
MONITOR_ID = "monitor_biglavgvq3"
response = requests.post(
f"{API_URL}/monitors/{MONITOR_ID}/pause",
headers={ "Authorization": f"Bearer {API_KEY}" },
)
print(response.status_code)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const monitorId = 'monitor_biglavgvq3'
const res = await fetch(`${API_URL}/monitors/${monitorId}/pause`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ' }
})
console.log(res.status)
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/monitors/monitor_biglavgvq3/pause" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
On success, returns `200` with the monitor and `status: paused`. `schedule.next_run_at` is `null` while paused.
## Resume a monitor
Resume a paused monitor with `POST /v1/monitors/:monitor_id/resume`.
Resuming re-enables the schedule and sets `status` back to `active`. Only `paused` monitors can be resumed.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
MONITOR_ID = "monitor_biglavgvq3"
response = requests.post(
f"{API_URL}/monitors/{MONITOR_ID}/resume",
headers={ "Authorization": f"Bearer {API_KEY}" },
)
print(response.status_code)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const monitorId = 'monitor_biglavgvq3'
const res = await fetch(`${API_URL}/monitors/${monitorId}/resume`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ' }
})
console.log(res.status)
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/monitors/monitor_biglavgvq3/resume" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
## Delete a monitor
Delete a monitor with `DELETE /v1/monitors/:monitor_id`.
Deletion soft-deletes the monitor row (`status: deleted`) and removes its schedule and shadow agent resources.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
MONITOR_ID = "monitor_biglavgvq3"
response = requests.delete(
f"{API_URL}/monitors/{MONITOR_ID}",
headers={ "Authorization": f"Bearer {API_KEY}" },
)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const monitorId = 'monitor_biglavgvq3'
const res = await fetch(`${API_URL}/monitors/${monitorId}`, {
method: 'DELETE',
headers: { 'Authorization': 'Bearer ' }
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X DELETE "https://api.olostep.com/v1/monitors/monitor_biglavgvq3" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
## Monitor status
| Status | Meaning |
| -------------- | ------------------------------------------------------------ |
| `provisioning` | Agent, spec, planner, and schedule are being set up |
| `active` | Schedule enabled; runs execute on `schedule.frequency` |
| `paused` | Schedule disabled via `/pause` |
| `failed` | Provisioning or schedule update failed (`error_message` set) |
| `deleted` | Soft-deleted via `DELETE` |
## Example use cases
Below are common monitor patterns. Each example only needs `query` and `frequency` at creation time; add `notification` and `webhook` later if you want alerts on the first run or on changes.
### Y Combinator new launches
Watch [Y Combinator Launches](https://www.ycombinator.com/launches/) for newly published startups. After planning, `tracked.type` is `urls` and `tracked.urls` points at the launches page.
```python Python theme={null}
import requests
API_URL = "https://api.olostep.com/v1"
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json",
}
requests.post(
f"{API_URL}/monitors",
headers=headers,
json={
"query": "Notify me when a new startup launches on Y Combinator Launches",
"frequency": "every 20 minutes",
},
)
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
await fetch(`${API_URL}/monitors`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
query: 'Notify me when a new startup launches on Y Combinator Launches',
frequency: 'every 20 minutes',
}),
})
```
```bash cURL theme={null}
curl -sS -X POST "https://api.olostep.com/v1/monitors" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "Notify me when a new startup launches on Y Combinator Launches",
"frequency": "every 20 minutes"
}'
```
Add email and webhook delivery after the monitor is `active`:
```bash theme={null}
curl -s -X POST "https://api.olostep.com/v1/monitors/monitor_biglavgvq3" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"notification": {
"channels": [{ "type": "email", "target": "you@example.com" }]
},
"webhook": { "url": "https://webhook.site/your-unique-id" }
}'
```
With `channels` set and `events` omitted, the API defaults to `["changed", "first_snapshot"]` so you are notified on the baseline run and whenever a change is detected.
### Competitor blog posts (AirOps, Profound)
Monitor a competitor blog index for new posts. The planner resolves `tracked.urls` to the blog URL (for example `https://www.airops.com/blog` or `https://www.tryprofound.com/blog`).
```bash cURL AirOps theme={null}
curl -sS -X POST "https://api.olostep.com/v1/monitors" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "Watch AirOps blog for new blog posts",
"frequency": "every hour"
}'
```
```bash cURL Profound theme={null}
curl -sS -X POST "https://api.olostep.com/v1/monitors" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "Watch Profound blog for new blog posts",
"frequency": "every 20 minutes"
}'
```
After creation, a monitor in this family looks like:
```json theme={null}
{
"id": "monitor_588ck513zd",
"object": "monitor",
"query": "Watch Profound blog for new blog posts",
"tracked": {
"type": "urls",
"urls": ["https://www.tryprofound.com/blog"],
"web_query": null
},
"schedule": {
"frequency": "every 20 minutes",
"cron": "15/20 * * * ? *",
"timezone": "UTC",
"next_run_at": "2026-05-29T15:15:00.000Z"
},
"status": "active"
}
```
Use `events: ["changed"]` on `notification` if you only want alerts when new posts appear, not on the first baseline snapshot.
### Stock price threshold (Tesla)
Monitor a structured data source when the condition is numeric rather than a page diff. The planner sets `tracked.type` to `data_api` and leaves `tracked.urls` empty.
```bash theme={null}
curl -sS -X POST "https://api.olostep.com/v1/monitors" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "Notify me when Tesla stock price drops below 436$",
"frequency": "every 12 minutes"
}'
```
```json theme={null}
{
"id": "monitor_7609p3191t",
"object": "monitor",
"query": "Notify me when Tesla stock price drops below 436$",
"tracked": {
"type": "data_api",
"urls": [],
"web_query": null
},
"schedule": {
"frequency": "every 12 minutes",
"cron": "2/12 * * * ? *",
"timezone": "UTC",
"next_run_at": "2026-05-29T15:02:00.000Z"
},
"status": "active"
}
```
### OpenAI API changelog
Get notified when [OpenAI’s API changelog](https://developers.openai.com/api/docs/changelog) lists new features, model releases, or deprecations. Mention the changelog URL in `query` or pin it with `source_policy.include_urls`.
```bash theme={null}
curl -sS -X POST "https://api.olostep.com/v1/monitors" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "Notify me when the OpenAI API changelog has new updates or features https://developers.openai.com/api/docs/changelog",
"frequency": "every hour",
"notification": {
"events": ["changed"],
"channels": [{ "type": "email", "target": "you@example.com" }]
}
}'
```
Set `events` to `["changed"]` so you are alerted when the changelog content changes, not only when the first snapshot is stored.
### Managing multiple monitors
List every monitor for your team to see status, schedules, and resolved targets in one place:
```bash theme={null}
curl -s -X GET "https://api.olostep.com/v1/monitors" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
A team running the examples above might see several monitors side by side—blog watches on different cadences, a `data_api` price watch, and a YC launches monitor with email and webhook configured—with `"count": 4` (or more) in the response.
## Common validation errors
The monitor endpoints return clear validation errors for common invalid requests:
* Missing or empty `query`
* `frequency` that is not scheduling language, resolves too often (under 10 minutes), or exceeds 50 characters
* Invalid `source_policy` entries (URL arrays must contain valid `http`/`https` strings)
* Invalid `notification` shape, unknown `events`, or invalid channel `type` / `target`
* Invalid `webhook.url` (must be `http` or `https`)
* Invalid `output_schema` (must be valid JSON Schema)
* Invalid `monitor_id` or `run_id` format
* Update while `status` is `provisioning` (`409`)
* Pause/resume when status is not `active` / `paused`
Example error:
```json theme={null}
{
"error": "Could not interpret 'frequency': \"check every second\". Use scheduling language such as \"every hour\" or \"every day at 9am\"."
}
```
# Schedules
Source: https://docs.olostep.com/features/schedules
Schedule API calls to run automatically at specified times
Through the Olostep `/v1/schedules` endpoint you can schedule API calls to execute automatically at specified times. Schedule one-time executions or recurring tasks using cron expressions or natural language.
* Schedule one-time executions at a specific datetime
* Create recurring schedules using cron expressions
* Use natural language text to automatically generate cron expressions
* Schedule HTTP endpoints (GET or POST)
* For POST requests, use short-form Olostep endpoints (automatically prefixed) or full URLs
* Pass any payload you want - the payload is sent exactly as you specify it
* Automatically manage schedule lifecycle
For API details see the [Schedule Endpoint API Reference](/api-reference/schedules/create).
## Installation
```python Python theme={null}
# pip install requests
import requests
```
```js Node theme={null}
// npm install node-fetch
// ESM
import fetch from 'node-fetch'
// CommonJS
const fetch = require('node-fetch')
```
```bash cURL theme={null}
# macOS: builtin curl is fine
```
## Create a schedule
Create a schedule to execute API calls automatically. You can create one-time schedules or recurring schedules using cron expressions. The `endpoint` can be any URL (not limited to Olostep endpoints), and the `payload` can contain any data you want to send.
### One-time schedule
Schedule an API call to execute once at a specific datetime.
```python Python theme={null}
import requests
import json
from datetime import datetime, timedelta
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
# Schedule a scrape to run in 1 hour
execute_at = (datetime.now() + timedelta(hours=1)).isoformat()
payload = {
"method": "POST",
"endpoint": "v1/scrapes",
"payload": {
"url_to_scrape": "https://example.com",
"formats": ["markdown"]
},
"execute_at": execute_at,
"expression_timezone": "UTC"
}
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
response = requests.post(f"{API_URL}/schedules", headers=headers, json=payload)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const execute_at = new Date(Date.now() + 60 * 60 * 1000).toISOString() // 1 hour from now
const res = await fetch(`${API_URL}/schedules`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
method: 'POST',
endpoint: 'v1/scrapes',
payload: {
url_to_scrape: 'https://example.com',
formats: ['markdown']
},
execute_at: execute_at,
expression_timezone: 'UTC'
})
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/schedules" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"method": "POST",
"endpoint": "v1/scrapes",
"payload": {
"url_to_scrape": "https://example.com",
"formats": ["markdown"]
},
"execute_at": "2025-01-15T10:00:00Z",
"expression_timezone": "UTC"
}'
```
### Recurring schedule with cron expression
Create a recurring schedule using a cron expression. Cron expressions use 6 fields format: minute hour day month day-of-week year.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
# Schedule a scrape to run every day at 10am UTC
payload = {
"method": "POST",
"endpoint": "v1/scrapes",
"payload": {
"url_to_scrape": "https://example.com",
"formats": ["markdown"]
},
"cron_expression": "0 10 * * ? *",
"expression_timezone": "UTC"
}
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
response = requests.post(f"{API_URL}/schedules", headers=headers, json=payload)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const res = await fetch(`${API_URL}/schedules`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
method: 'POST',
endpoint: 'v1/scrapes',
payload: {
url_to_scrape: 'https://example.com',
formats: ['markdown']
},
cron_expression: '0 10 * * ? *', // Every day at 10am UTC
expression_timezone: 'UTC'
})
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/schedules" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"method": "POST",
"endpoint": "v1/scrapes",
"payload": {
"url_to_scrape": "https://example.com",
"formats": ["markdown"]
},
"cron_expression": "0 10 * * ? *",
"expression_timezone": "UTC"
}'
```
### Natural language scheduling
Use natural language text to automatically generate cron expressions. The system will convert your text into a valid cron expression.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
# Schedule using natural language
payload = {
"method": "POST",
"endpoint": "v1/scrapes",
"payload": {
"url_to_scrape": "https://example.com",
"formats": ["markdown"]
},
"text": "every 3 minutes",
"expression_timezone": "UTC"
}
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
response = requests.post(f"{API_URL}/schedules", headers=headers, json=payload)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const res = await fetch(`${API_URL}/schedules`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
method: 'POST',
endpoint: 'v1/scrapes',
payload: {
url_to_scrape: 'https://example.com',
formats: ['markdown']
},
text: 'every Monday at 9am',
expression_timezone: 'UTC'
})
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/schedules" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"method": "POST",
"endpoint": "v1/scrapes",
"payload": {
"url_to_scrape": "https://example.com",
"formats": ["markdown"]
},
"text": "every day at 10am",
"expression_timezone": "UTC"
}'
```
## Response format
When you create a schedule, you'll receive a schedule object with the following properties:
```json theme={null}
{
"id": "schedule_abc123xyz",
"object": "schedule"
"type": "recurring",
"method": "POST",
"endpoint": "v1/scrapes",
"cron_expression": "0 10 * * ? *",
"expression_timezone": "UTC",
"created": "2025-01-15T10:00:00.000Z"
}
```
For one-time schedules, the response includes `execute_at` instead of `cron_expression`:
```json theme={null}
{
"id": "schedule_abc123xyz",
"object": "schedule"
"type": "onetime",
"method": "POST",
"endpoint": "v1/scrapes",
"execute_at": "2025-01-15T10:00:00.000Z",
"expression_timezone": "UTC",
"created": "2025-01-15T09:00:00.000Z"
}
```
## List schedules
Retrieve all schedules for your team. By default, deleted schedules are filtered out. Use the `include_deleted` query parameter to include them.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
headers = {
"Authorization": f"Bearer {API_KEY}"
}
response = requests.get(f"{API_URL}/schedules", headers=headers)
result = response.json()
print(f"Total schedules: {result['count']}")
for schedule in result['schedules']:
print(json.dumps(schedule, indent=2))
# To include deleted schedules:
# response = requests.get(f"{API_URL}/schedules?include_deleted=true", headers=headers)
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const res = await fetch(`${API_URL}/schedules`, {
headers: { 'Authorization': 'Bearer ' }
})
const result = await res.json()
console.log(`Total schedules: ${result.count}`)
result.schedules.forEach(s => console.log(s))
// To include deleted schedules:
// const res = await fetch(`${API_URL}/schedules?include_deleted=true`, { ... })
```
```bash cURL theme={null}
curl -s -X GET "https://api.olostep.com/v1/schedules" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
# To include deleted schedules:
# curl -s -X GET "https://api.olostep.com/v1/schedules?include_deleted=true" \
# -H "Authorization: Bearer $OLOSTEP_API_KEY"
```
## Get a schedule
Retrieve a single schedule by its ID.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
schedule_id = "schedule_abc123xyz"
headers = {
"Authorization": f"Bearer {API_KEY}"
}
response = requests.get(f"{API_URL}/schedules/{schedule_id}", headers=headers)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const scheduleId = 'schedule_abc123xyz'
const res = await fetch(`${API_URL}/schedules/${scheduleId}`, {
headers: { 'Authorization': 'Bearer ' }
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X GET "https://api.olostep.com/v1/schedules/schedule_abc123xyz" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
## Delete a schedule
Delete a schedule by its ID. This will stop any future executions.
```python Python theme={null}
import requests
import json
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
schedule_id = "schedule_abc123xyz"
headers = {
"Authorization": f"Bearer {API_KEY}"
}
response = requests.delete(f"{API_URL}/schedules/{schedule_id}", headers=headers)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const scheduleId = 'schedule_abc123xyz'
const res = await fetch(`${API_URL}/schedules/${scheduleId}`, {
method: 'DELETE',
headers: { 'Authorization': 'Bearer ' }
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X DELETE "https://api.olostep.com/v1/schedules/schedule_abc123xyz" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
## Supported endpoints
### Olostep endpoints (short form)
For POST requests, you can use short forms for Olostep endpoints. The system will automatically prepend `https://api.olostep.com/` for these:
* `v1/scrapes` - Schedule web scraping tasks
* `v1/batches` - Schedule batch processing jobs
* `v1/crawls` - Schedule website crawling operations
* `v1/maps` - Schedule map data extraction
* `v1/answers` - Schedule answer generation
### Full URLs
You can also provide full URLs for your endpoints. This is required for external APIs or webhooks:
```python Python theme={null}
import requests
import json
from datetime import datetime, timedelta
API_KEY = ""
API_URL = "https://api.olostep.com/v1"
# Schedule a call to an external API
payload = {
"method": "POST",
"endpoint": "https://api.example.com/webhook",
"payload": {
"custom_field": "any value",
"data": {"nested": "structure"},
"timestamp": datetime.now().isoformat()
},
"execute_at": (datetime.now() + timedelta(hours=1)).isoformat(),
"expression_timezone": "UTC"
}
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
response = requests.post(f"{API_URL}/schedules", headers=headers, json=payload)
print(json.dumps(response.json(), indent=2))
```
```js Node theme={null}
const API_URL = 'https://api.olostep.com/v1'
const execute_at = new Date(Date.now() + 60 * 60 * 1000).toISOString()
const res = await fetch(`${API_URL}/schedules`, {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
method: 'POST',
endpoint: 'https://api.example.com/webhook',
payload: {
custom_field: 'any value',
data: { nested: 'structure' },
timestamp: new Date().toISOString()
},
execute_at: execute_at,
expression_timezone: 'UTC'
})
})
console.log(await res.json())
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/schedules" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"method": "POST",
"endpoint": "https://api.example.com/webhook",
"payload": {
"custom_field": "any value",
"data": {"nested": "structure"}
},
"execute_at": "2025-01-15T10:00:00Z",
"expression_timezone": "UTC"
}'
```
The `payload` field accepts any JSON object - you can structure it however you need for your target endpoint.
## Cron expression format
Cron expressions use 6 fields format:
```
minute hour day month day-of-week year
```
Examples:
* `0/3 * * * ? *` - Every 3 minutes
* `0 10 * * ? *` - Every day at 10:00 AM
* `0 9 ? * MON *` - Every Monday at 9:00 AM
* `0 0 1 * ? *` - First day of every month at midnight
Use `?` for day-of-month or day-of-week when not specified.
## Natural language examples
You can use natural language to describe schedules. The system will automatically convert them to cron expressions:
* "every 3 minutes" → `0/3 * * * ? *`
* "every day at 10am" → `0 10 * * ? *`
* "every Monday at 9am" → `0 9 ? * MON *`
* "every hour" → `0 * * * ? *`
* "every week on Monday" → `0 0 ? * MON *`
## Important notes
* One-time schedules are automatically deleted after execution
* Recurring schedules continue until manually deleted
* Timezone must be a valid IANA timezone identifier (e.g., "UTC", "America/New\_York", "Europe/London")
* The `execute_at` datetime must be in the future
* Natural language conversion may require retries; the system will attempt up to 3 times
* When using natural language text (`text` parameter), the timezone defaults to "UTC"
* Schedules execute the API call with the provided payload exactly as specified - you can pass any JSON structure you need
* For POST requests, short-form Olostep endpoints (`v1/scrapes`, `v1/batches`, `v1/crawls`, `v1/maps`, `v1/answers`) are automatically prefixed with `https://api.olostep.com/`
* For other endpoints, provide the full URL
* The `payload` can contain any data structure - it's sent as-is to your target endpoint
* Deleting an already deleted schedule will return a 400 error
## Pricing
Schedules themselves are free. You only pay for the API calls that are executed when the schedule runs. For example, if you schedule a scrape, you'll be charged 1 credit per execution (or more if using parsers or LLM extraction).
# Scrape
Source: https://docs.olostep.com/features/scrapes
Turn any URL into LLM-ready Markdown, HTML, screenshots, PDFs, or structured JSON.
Through the Olostep `/v1/scrapes` endpoint you can extract LLM-friendly Markdown, HTML, text, screenshots, or structured JSON from any URL in real time.
* Outputs clean markdown, structured data, screenshots, or html
* Extract JSON through [Parsers](/features/structured-content/parsers) or [LLM extraction](/features/structured-content/llm-extraction)
* Handles dynamic content: js-rendered sites, login flows via actions, PDFs
For API details see the [Scrape Endpoint API Reference](/api-reference/scrapes/create).
## Scraping a URL
Use the `/v1/scrapes` endpoint to scrape a single URL and choose output formats.
### Installation
```python Python theme={null}
pip install olostep
```
```javascript Node theme={null}
npm install olostep
```
```bash cURL theme={null}
# curl is available by default on macOS, Linux, and Windows
```
```javascript Node (API) theme={null}
npm install node-fetch
```
```bash Python (API) theme={null}
pip install requests
```
### Usage
You can use the endpoint to scrape a single URL and choose output formats. The mandatory parameters are `url_to_scrape` and `formats`.
Some other common parameters are `wait_before_scraping` (in milliseconds), `remove_css_selectors` (default, none, or an array of selectors), and `country`.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
result = client.scrapes.create(
url_to_scrape="https://en.wikipedia.org/wiki/Alexander_the_Great",
formats=["markdown", "html"],
)
print(result.markdown_content)
print(result.html_content)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const result = await client.scrapes.create({
url: 'https://en.wikipedia.org/wiki/Alexander_the_Great',
formats: ['markdown', 'html'],
})
console.log(result.markdown_content)
console.log(result.html_content)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/scrapes" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url_to_scrape": "https://en.wikipedia.org/wiki/Alexander_the_Great",
"formats": ["markdown", "html"]
}'
```
```bash CLI theme={null}
olostep scrape "https://en.wikipedia.org/wiki/Alexander_the_Great" \
--formats markdown,html
```
```js Node (API) theme={null}
const endpoint = 'https://api.olostep.com/v1/scrapes'
const payload = {
url_to_scrape: 'https://en.wikipedia.org/wiki/Alexander_the_Great',
formats: ['markdown', 'html']
}
const res = await fetch(endpoint, {
method: 'POST',
headers: {
'Authorization': 'Bearer ',
'Content-Type': 'application/json'
},
body: JSON.stringify(payload)
})
const data = await res.json()
console.log(data)
```
```python Python (API) theme={null}
import requests
import json
endpoint = "https://api.olostep.com/v1/scrapes"
payload = {
"url_to_scrape": "https://en.wikipedia.org/wiki/Alexander_the_Great",
"formats": ["markdown", "html"]
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
response = requests.post(endpoint, json=payload, headers=headers)
print(json.dumps(response.json(), indent=2))
```
### Response
The API returns a `scrape` object in response.
The `scrape` has a few properties like `id` and `result`.
The `result` object has the following fields (according to the `formats` parameter some might be null):
* `html_content`: the HTML content of the page. Pass `formats: ["html"]` to get this.
* `markdown_content`: the MD content of the page. Pass `formats: ["markdown"]` to get this.
* `text_content`: the text content of the page. Pass `formats: ["text"]` to get this.
* `json_content`: the JSON content of the page. Pass `formats: ["json"]` to get this and also provide a `parser` or `llm_extract` parameter.
* `screenshot_hosted_url`: the hosted URL of the screenshot.
* `html_hosted_url`: the hosted URL of the HTML content
* `markdown_hosted_url`: the hosted URL of the Markdown content
* `json_hosted_url`: the hosted URL of the JSON content
* `text_hosted_url`: the hosted URL of the text content
* `links_on_page`: the links on the page
* `page_metadata`: the metadata of the page
```json theme={null}
{
"id": "scrape_6h89o8u1kt",
"object": "scrape",
"created": 1745673871,
"metadata": {},
"retrieve_id": "6h89o8u1kt",
"url_to_scrape": "https://en.wikipedia.org/wiki/Alexander_the_Great",
"result": {
"html_content": "
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
# Opt-in to caching: Accept results up to 1 day (86400 seconds) old
result = client.scrapes.create(
url_to_scrape="https://example.com",
formats=["markdown"],
max_age=86400
)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
// Opt-in to caching: Accept results up to 1 day (86400 seconds) old
const result = await client.scrapes.create({
url: 'https://example.com',
formats: ['markdown'],
maxAge: 86400,
})
```
```bash cURL theme={null}
# Opt-in to caching: Accept results up to 1 hour (3600 seconds) old
curl -X POST "https://api.olostep.com/v1/scrapes" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url_to_scrape": "https://example.com",
"formats": ["markdown"],
"max_age": 3600
}'
```
```js Node (API) theme={null}
const endpoint = 'https://api.olostep.com/v1/scrapes'
const payload = {
url_to_scrape: 'https://example.com',
formats: ['markdown'],
max_age: 86400 // Accept results up to 1 day (86400 seconds) old
}
const res = await fetch(endpoint, {
method: 'POST',
headers: {
'Authorization': 'Bearer ',
'Content-Type': 'application/json'
},
body: JSON.stringify(payload)
})
const data = await res.json()
console.log(data)
```
```python Python (API) theme={null}
import requests
import json
endpoint = "https://api.olostep.com/v1/scrapes"
payload = {
"url_to_scrape": "https://example.com",
"formats": ["markdown"],
"max_age": 86400 # Accept results up to 1 day (86400 seconds) old
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
response = requests.post(endpoint, json=payload, headers=headers)
print(json.dumps(response.json(), indent=2))
```
### When is the cache skipped?
The cache is automatically bypassed (forcing a live scrape) when your request needs:
* **Interactive sessions:** Requests using `session_id` or loading a custom browser `context`.
* **Screenshots:** Any request that includes `screenshot` in formats or sets the screenshot option bypasses the cache.
* **Special file types:** Binary file downloads or raw PDF rendering.
* **Debugging & Network:** Capturing `network_calls` or using async parser jobs.
## Extracting links
Pass a `links_on_page` object in the request to collect the links found on the page. All links are returned as absolute URLs.
```json theme={null}
"links_on_page": {
"include_links": ["/blog/*"],
"exclude_links": ["*.pdf"],
"query_to_order_links_by": "pricing"
}
```
* `include_links` / `exclude_links`: glob patterns matched against each link's URL **path**.
* `query_to_order_links_by`: re-orders the returned links by relevance to this text.
Glob patterns match path segments. A single `*` does **not** cross `/`, so `"/blog/*"` matches `"/blog/post-1"` but **not** the index `"/blog"` itself — and it never matches `"/blog?tag=x"` because query strings are not part of the path. To include the index too, use `"/blog*"` or `"{/blog,/blog/**}"`.
## Scrape Formats
Choose one or more output formats via `formats`:
* `markdown`: LLM-friendly markdown
* `html`: cleaned HTML
* `text`: plain text
* `json`: structured output (via parser or llm\_extract)
* `raw_pdf`: raw PDF bytes extracted to hosted URL
* `screenshot`: set via actions to capture a screenshot and return a hosted URL
Output keys are returned inside `result` as `*_content` fields and a `*_hosted_url` as well.
## Extract structured data
You can extract structured JSON in two ways: using Parsers or LLM extraction.
### Using a Parser (recommended for scale)
Define `formats: ["json"]` and provide a parser `id`.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
result = client.scrapes.create(
url_to_scrape="https://www.google.com/search?q=alexander+the+great&gl=us&hl=en",
formats=["json"],
parser="@olostep/google-search",
)
print(result.json_content)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const result = await client.scrapes.create({
url: 'https://www.google.com/search?q=alexander+the+great&gl=us&hl=en',
formats: ['json'],
parser: '@olostep/google-search',
})
console.log(result.json_content)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/scrapes" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url_to_scrape": "https://www.google.com/search?q=alexander+the+great&gl=us&hl=en",
"formats": ["json"],
"parser": {"id": "@olostep/google-search"}
}'
```
```bash CLI theme={null}
olostep scrape "https://www.google.com/search?q=alexander+the+great&gl=us&hl=en" \
--formats json \
--payload-json '{"parser":{"id":"@olostep/google-search"}}'
```
```js Node (API) theme={null}
const res = await fetch('https://api.olostep.com/v1/scrapes', {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
url_to_scrape: 'https://www.google.com/search?q=alexander+the+great&gl=us&hl=en',
formats: ['json'],
parser: { id: '@olostep/google-search' }
})
})
console.log(await res.json())
```
```python Python (API) theme={null}
import requests, json
endpoint = "https://api.olostep.com/v1/scrapes"
payload = {
"url_to_scrape": "https://www.google.com/search?q=alexander+the+great&gl=us&hl=en",
"formats": ["json"],
"parser": {
"id": "@olostep/google-search"
}
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
res = requests.post(endpoint, json=payload, headers=headers)
print(json.dumps(res.json(), indent=2))
```
Olostep has a few pre-built parsers for [popular websites](https://www.olostep.com/store) but you can also create your own parsers through the dashboard or ask our team to do it for you.
Parsers are self-healing and will update themselves to the latest version of the website.
### Using LLM extraction (schema and/or prompt)
Provide `llm_extract` with a JSON Schema (`schema`) and/or a natural language instruction (`prompt`). You can pass both parameters, but if both are provided, `schema` takes precedence.
Instead, if you just pass a `prompt`, the LLM will extract the data based on the prompt and will decide the data structure on its own.
```python Python theme={null}
from olostep import LLMExtract, Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
result = client.scrapes.create(
url_to_scrape="https://www.berklee.edu/events/stefano-marchese-friends",
formats=["markdown", "json"],
llm_extract=LLMExtract(
schema={
"event": {
"type": "object",
"properties": {
"title": {"type": "string"},
"date": {"type": "string"},
"description": {"type": "string"},
"venue": {"type": "string"},
"address": {"type": "string"},
"start_time": {"type": "string"},
},
}
}
),
)
print(result.json_content)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const result = await client.scrapes.create({
url: 'https://www.berklee.edu/events/stefano-marchese-friends',
formats: ['markdown', 'json'],
llmExtract: {
schema: {
event: {
type: 'object',
properties: {
title: { type: 'string' },
date: { type: 'string' },
description: { type: 'string' },
venue: { type: 'string' },
address: { type: 'string' },
start_time: { type: 'string' },
},
},
},
},
})
console.log(result.json_content)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/scrapes" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url_to_scrape": "https://www.berklee.edu/events/stefano-marchese-friends",
"formats": ["json"],
"llm_extract": {
"prompt": "Extract the event title, date, description, venue, address, and start time from the page."
}
}'
```
```bash CLI theme={null}
olostep scrape "https://www.berklee.edu/events/stefano-marchese-friends" \
--formats json \
--payload-json '{"llm_extract":{"prompt":"Extract the event title, date, description, venue, address, and start time from the page."}}'
```
```js Node (API) theme={null}
const res = await fetch('https://api.olostep.com/v1/scrapes', {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
url_to_scrape: 'https://www.berklee.edu/events/stefano-marchese-friends',
formats: ['json'],
llm_extract: {
prompt: 'Extract the event title, date, description, venue, address, and start time from the page.'
}
})
})
console.log(await res.json())
```
```python Python (API) theme={null}
import requests, json
endpoint = "https://api.olostep.com/v1/scrapes"
payload = {
"url_to_scrape": "https://www.berklee.edu/events/stefano-marchese-friends",
"formats": ["markdown", "json"],
"llm_extract": {
"schema": {
"event": {
"type": "object",
"properties": {
"title": {"type": "string"},
"date": {"type": "string"},
"description": {"type": "string"},
"venue": {"type": "string"},
"address": {"type": "string"},
"start_time": {"type": "string"}
}
}
}
}
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
res = requests.post(endpoint, json=payload, headers=headers)
print(json.dumps(res.json(), indent=2))
```
Note: `result.json_content` returns a stringified JSON. Parse it in your code if you need an object.
**Pricing:** `llm_extract` costs 10 credits per scrape. To lower the cost, you can bring your own API keys or enable usage-based pricing. Contact [info@olostep.com](mailto:info@olostep.com) to get access.
## Extract links on the page
With the `links_on_page` option, you can extract all the links present on the page you scrape. It accepts the following parameters to help filter and order the extracted links:
* `absolute_links` (boolean, default: `true`): When true, it returns complete URLs (e.g., `https://example.com/page`) instead of relative paths (e.g., `/page`).
* `query_to_order_links_by` (string): Orders the returned links by their similarity to the provided query text, prioritizing the most relevant matches first.
* `include_links` (array of strings): Filter extracted links using glob patterns. Use patterns like `*.pdf` to match file extensions, `/blog/*` for specific paths, or full URLs like `https://example.com/*`. Supports wildcards (`*`), character classes (`[a-z]`), and alternation (`{pattern1,pattern2}`).
* `exclude_links` (array of strings): Exclude specific links using glob patterns, following the same syntax as `include_links`.
## Interacting with the page with Actions
Perform actions before scraping to interact with dynamic sites. Supported actions:
* `wait` with `milliseconds`
* `click` with `selector`
* `fill_input` with `selector` and `value`
* `scroll` with `direction` and `amount`
It is often useful to use `wait` before/after other actions to allow the page to load.
### Example
```python Python theme={null}
from olostep import FillInputAction, Olostep, WaitAction
client = Olostep(api_key="YOUR_REAL_KEY")
result = client.scrapes.create(
url_to_scrape="https://example.com/login",
formats=["markdown"],
actions=[
FillInputAction(selector="input[type=email]", value="john@example.com"),
WaitAction(milliseconds=500),
FillInputAction(selector="input[type=password]", value="secret"),
{"type": "click", "selector": "button[type=\"submit\"]"},
WaitAction(milliseconds=1500),
],
)
print(result.markdown_content)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const result = await client.scrapes.create({
url: 'https://example.com/login',
formats: ['markdown'],
actions: [
{ type: 'fill_input', selector: 'input[type=email]', value: 'john@example.com' },
{ type: 'wait', milliseconds: 500 },
{ type: 'fill_input', selector: 'input[type=password]', value: 'secret' },
{ type: 'click', selector: 'button[type="submit"]' },
{ type: 'wait', milliseconds: 1500 },
],
})
console.log(result.markdown_content)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/scrapes" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url_to_scrape": "https://example.com/login",
"formats": ["markdown"],
"actions": [
{ "type": "fill_input", "selector": "input[type=email]", "value": "john@example.com" },
{ "type": "wait", "milliseconds": 500 },
{ "type": "fill_input", "selector": "input[type=password]", "value": "secret" },
{ "type": "click", "selector": "button[type=\"submit\"]" },
{ "type": "wait", "milliseconds": 1500 }
]
}'
```
```bash CLI theme={null}
# For complex options like actions, use --payload-file with a JSON file
olostep scrape "https://example.com/login" \
--formats markdown \
--payload-file actions.json
# Where actions.json contains:
# {
# "actions": [
# {"type": "fill_input", "selector": "input[type=email]", "value": "john@example.com"},
# {"type": "wait", "milliseconds": 500},
# {"type": "fill_input", "selector": "input[type=password]", "value": "secret"},
# {"type": "click", "selector": "button[type=\"submit\"]"},
# {"type": "wait", "milliseconds": 1500}
# ]
# }
```
```js Node (API) theme={null}
const res = await fetch('https://api.olostep.com/v1/scrapes', {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
url_to_scrape: 'https://example.com/login',
formats: ['markdown'],
actions: [
{ type: 'fill_input', selector: 'input[type=email]', value: 'john@example.com' },
{ type: 'wait', milliseconds: 500 },
{ type: 'fill_input', selector: 'input[type=password]', value: 'secret' },
{ type: 'click', selector: 'button[type="submit"]' },
{ type: 'wait', milliseconds: 1500 }
]
})
})
console.log(await res.json())
```
```python Python (API) theme={null}
import requests, json
endpoint = "https://api.olostep.com/v1/scrapes"
payload = {
"url_to_scrape": "https://example.com/login",
"formats": ["markdown"],
"actions": [
{"type": "fill_input", "selector": "input[type=email]", "value": "john@example.com"},
{"type": "wait", "milliseconds": 500},
{"type": "fill_input", "selector": "input[type=password]", "value": "secret"},
{"type": "click", "selector": "button[type=\"submit\"]"},
{"type": "wait", "milliseconds": 1500}
]
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
res = requests.post(endpoint, json=payload, headers=headers)
print(json.dumps(res.json(), indent=2))
```
The response will include any requested formats (e.g., `markdown_content`).
## Use Cases
Below are a few practical applications of customers using the `/scrapes` endpoint.
### Content Analysis & Research
* **Competitive Analysis**: Extract product details, pricing, and features from competitor websites
* **Market Research**: Analyze landing pages, product descriptions, and customer testimonials
* **Academic Research**: Gather specific data from scientific publications or research portals
* **Legal Documentation**: Extract case studies, regulations, or legal precedents from official websites
### E-commerce & Retail
* **Dynamic Pricing Strategies**: Get real-time product pricing from competing stores
* **Product Information Management**: Extract detailed specifications and descriptions
* **Stock/Inventory Monitoring**: Check product availability at other retailers
* **Review Analysis**: Gather consumer feedback and sentiment for specific products
### Marketing & Content Creation
* **Content Curation**: Extract relevant articles and blog posts for newsletters
* **SEO Analysis**: Examine competitors' keyword usage, meta descriptions, and page structure
* **Lead Generation**: Extract contact information from business directories or company pages
* **Influencer Research**: Gather engagement metrics and content styles from influencer profiles
* **Personalised Social Media generation**: Create AI-powered social media marketing by analyzing customers websites
### Data Applications
* **AI Training Data Collection**: Gather specific examples for machine learning models
* **Custom Knowledge Base Building**: Extract documentation or instructions from software sites
* **Historical Data Archives**: Preserve website content at specific points in time
* **Structured Data Extraction**: Transform web content into formatted datasets for analysis
### Monitoring & Alerts
* **Regulatory Compliance Monitoring**: Track changes to legal or regulatory websites
* **Crisis Management**: Monitor news sites for mentions of specific events or organizations
* **Event Tracking**: Extract details about upcoming events from venue or organizer websites
* **Service Status Monitoring**: Check service status pages for specific platforms or tools
### Publishing & Media
* **News Aggregation**: Extract breaking news from official sources
* **Media Monitoring**: Track specific topics across news sites
* **Content Verification**: Extract information to fact-check claims or statements
* **Multimedia Extraction**: Gather embedded videos, images, or audio for media libraries
### Financial Applications
* **Investment Research**: Extract financial statements or annual reports from company websites
* **Economic Indicators**: Gather economic data from government or financial institution websites
* **Cryptocurrency Data**: Extract real-time pricing and market cap information
* **Financial News Analysis**: Monitor financial news sites for specific market signals
### Technical Applications
* **API Documentation Extraction**: Gather technical documentation for reference
* **Integration Testing**: Extract website elements to verify third-party integrations
* **Accessibility Testing**: Analyze website structure for compliance with accessibility standards
* **Web Archive Creation**: Capture full website content for historical preservation
### Integration Scenarios
* **CRM Systems**: Enhance customer profiles with data from company websites or Linkedin
* **Content Management Systems**: Import relevant external content
* **Business Intelligence Tools**: Supplement internal data with external market information
* **Project Management Software**: Extract specifications or requirements from client websites
* **Custom Dashboards**: Display extracted data alongside internal metrics
## Error Handling
All errors follow a shared envelope shape. Check `error.type` and `error.code` to branch programmatically:
```json theme={null}
{
"id": "error_abc123",
"object": "error",
"created": 1745673871,
"url": "https://example.com",
"metadata": {},
"error": {
"type": "...",
"code": "...",
"message": "..."
}
}
```
| HTTP | `error.type` | `error.code` | Meaning |
| ---- | ----------------------- | ----------------------- | -------------------------------------------------------------------------------------------------------------- |
| 400 | `invalid_request_error` | `dns_resolution_failed` | The domain does not exist or the URL has a typo. |
| 400 | `invalid_request_error` | `invalid_url` | The URL is malformed. |
| 502 | `invalid_request_error` | `tls_error` | The website has an invalid or incompatible TLS/SSL certificate. `error.detail` carries the low-level SSL code. |
| 504 | `request_timeout` | `scrape_poll_timeout` | The scrape did not finish within the \~55-second wait budget. |
### DNS failure (400)
The domain does not resolve. Check the URL for typos.
```json theme={null}
{
"error": {
"type": "invalid_request_error",
"code": "dns_resolution_failed",
"message": "The URL contains a typo, or the domain does not exist."
}
}
```
### TLS/SSL error (502)
The target website has a broken or incompatible HTTPS configuration. `error.detail` provides the specific SSL error code for diagnostics; `error.code` is always `tls_error`.
```json theme={null}
{
"error": {
"type": "invalid_request_error",
"code": "tls_error",
"detail": "err_ssl_tlsv1_alert_internal_error",
"message": "The website closed or rejected the TLS handshake. The server may be misconfigured or use an unsupported SSL/TLS version."
}
}
```
### Request timeout (504)
The scrape did not complete within the wait budget. The page may be slow, bot-protected, or temporarily unavailable. This response is safe to retry.
```json theme={null}
{
"error": {
"type": "request_timeout",
"code": "scrape_poll_timeout",
"message": "Request timed out while waiting for scrape result. The page may be slow, blocked for our fetchers, or temporarily unavailable."
}
}
```
## Pricing
Scrape costs 1 credit by default. If you also pass [parsers](/features/structured-content/parsers), the costs vary by parser (1-5 credits). If you use [LLM extract](/features/structured-content/llm-extraction), it costs 10 credits.
# Search API
Source: https://docs.olostep.com/features/search
Search the web with natural language and get structured links — Olostep's AI search API.
The Olostep `/v1/searches` endpoint lets you search the web with a natural language query and get back a deduplicated list of relevant links with titles and descriptions.
* Send a query in plain English
* Get back structured links from across the web
* Optionally scrape every returned URL in one round-trip and embed `markdown_content` / `html_content` directly into the response
* Filter by domain, control the result count, and bound the scraping wallclock
It will search for the query semantically across the web and return results.
For API details, see the [Search Endpoint API Reference](/api-reference/searches/create).
## Installation
```python Python theme={null}
pip install olostep
```
```javascript Node theme={null}
npm install olostep
```
```bash cURL theme={null}
# curl is available by default on macOS, Linux, and Windows
```
```javascript Node (API) theme={null}
npm install node-fetch
```
```bash Python (API) theme={null}
pip install requests
```
## Basic usage
Send a natural language query and receive a list of relevant links.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
search = client.searches.create("Best Answer Engine Optimization startups")
print(search.id, len(search.links))
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const search = await client.searches.create('Best Answer Engine Optimization startups')
console.log(search.id, search.links.length)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/searches" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "Best Answer Engine Optimization startups"
}'
```
```js Node (API) theme={null}
const res = await fetch('https://api.olostep.com/v1/searches', {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
query: 'Best Answer Engine Optimization startups'
})
})
console.log(await res.json())
```
```python Python (API) theme={null}
import requests, json
endpoint = "https://api.olostep.com/v1/searches"
payload = {
"query": "Best Answer Engine Optimization startups"
}
headers = {"Authorization": "Bearer ", "Content-Type": "application/json"}
response = requests.post(endpoint, json=payload, headers=headers)
print(json.dumps(response.json(), indent=2))
```
## Request parameters
| Field | Type | Required | Default | Description |
| ----------------- | --------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `query` | string | yes | — | The search query in natural language. |
| `limit` | integer | no | `12` | Maximum number of links to return after deduplication. Must be between `1` and `25`. |
| `include_domains` | string\[] | no | `[]` | Restrict results to these domains. Bare hosts only — leading `http(s)://` and trailing slashes are stripped automatically. |
| `exclude_domains` | string\[] | no | `[]` | Exclude results from these domains. Bare hosts only — leading `http(s)://` and trailing slashes are stripped automatically. |
| `scrape_options` | object | no | — | When provided, every returned link is also scraped and its content embedded in the response. See [scrape\_options](#scrape-options) below. |
| `fast_mode` | boolean | no | `false` | Request a direct, low-latency search using your query verbatim. Default mode performs a broader search pass for wider result coverage. |
### Limiting the number of results
```json theme={null}
{
"query": "What's going on with OpenAI's Sora shutting down?",
"limit": 5
}
```
### Filtering by domain
`include_domains` narrows results to a whitelist; `exclude_domains` filters out unwanted sources. They can be combined.
```json theme={null}
{
"query": "OpenAI Sora shutdown analysis",
"include_domains": ["nytimes.com", "wsj.com", "bbc.com"],
"exclude_domains": ["pinterest.com"]
}
```
## scrape\_options
Pass `scrape_options` to scrape every returned URL in parallel and embed the rendered content directly on each link. This saves a round-trip per result vs. calling `/v1/searches` and `/v1/scrapes` separately.
```json theme={null}
{
"query": "What's going on with OpenAI's Sora shutting down?",
"limit": 10,
"scrape_options": {
"formats": ["markdown"],
"remove_css_selectors": "default",
"timeout": 25
}
}
```
| Field | Type | Default | Description |
| ---------------------- | --------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `formats` | string\[] | `["markdown"]` | Output formats to attach to each link. For `/v1/searches`, only `"html"` and `"markdown"` are supported. Pass `["html", "markdown"]` to receive both. |
| `remove_css_selectors` | string | `"default"` | Forwarded to `/v1/scrapes`. `"default"` strips nav/footer/script/style/svg/dialog noise. Use `"none"` to disable, or pass a JSON-stringified array of selectors to remove. |
| `timeout` | integer | `25` | Wallclock budget in **seconds** for the entire scrape phase. Must be between `1` and `60`. After this elapses, the search returns immediately — content fields will be `null` for any links that hadn't finished. |
### Behavior
* All links are scraped **in parallel**. The `timeout` bounds the whole batch, not each individual link.
* Per-link scrape failures (network errors, individual page timeouts) leave that link's `markdown_content` / `html_content` as `null` while other links return normally.
* If the global `timeout` elapses before all scrapes finish, the search responds immediately with the links it has — already-completed scrapes keep their content; in-flight ones come back with `null` content.
* For `reddit.com/.../comments/...` URLs, the request is automatically routed through the `@olostep/reddit-post` parser and the structured JSON is rendered into clean markdown + basic HTML
* If the combined inline content exceeds 9MB, content fields are nulled, `result.size_exceeded` is set to `true`, and you can fetch the full payload from `result.json_hosted_url`.
### Example with scraping
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
search = client.searches.create(
query="What's going on with OpenAI's Sora shutting down?",
limit=5,
scrape_options={"formats": ["markdown"], "timeout": 25},
)
for link in search.links:
print(link["url"], "—", len(link.get("markdown_content") or ""), "chars")
```
```js Node theme={null}
import Olostep, { Format } from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const search = await client.searches.create({
query: "What's going on with OpenAI's Sora shutting down?",
limit: 5,
scrapeOptions: {
formats: [Format.MARKDOWN],
timeout: 25
}
})
for (const link of search.links) {
console.log(link.url, '—', (link.markdown_content || '').length, 'chars')
}
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/searches" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"query": "What'"'"'s going on with OpenAI'"'"'s Sora shutting down?",
"limit": 5,
"scrape_options": {
"formats": ["markdown"],
"timeout": 25
}
}'
```
```js Node (API) theme={null}
const res = await fetch('https://api.olostep.com/v1/searches', {
method: 'POST',
headers: { 'Authorization': 'Bearer ', 'Content-Type': 'application/json' },
body: JSON.stringify({
query: "What's going on with OpenAI's Sora shutting down?",
limit: 5,
scrape_options: {
formats: ['markdown'],
timeout: 25
}
})
})
const data = await res.json()
for (const link of data.result.links) {
console.log(link.url, '—', (link.markdown_content || '').length, 'chars')
}
```
```python Python (API) theme={null}
import requests, json
endpoint = "https://api.olostep.com/v1/searches"
payload = {
"query": "What's going on with OpenAI's Sora shutting down?",
"limit": 5,
"scrape_options": {
"formats": ["markdown"],
"timeout": 25
}
}
headers = {"Authorization": "Bearer ", "Content-Type": "application/json"}
response = requests.post(endpoint, json=payload, headers=headers)
data = response.json()
for link in data["result"]["links"]:
print(link["url"], "—", len(link.get("markdown_content") or ""), "chars")
```
## Response
You will receive a `search` object in response. The `search` object contains an `id`, your original `query`, `credits_consumed`, and a `result` with a list of `links`.
```json theme={null}
{
"id": "search_9bi0sbj9xa",
"object": "search",
"created": 1760327323,
"metadata": {},
"query": "What's going on with OpenAI's Sora shutting down?",
"credits_consumed": 10,
"result": {
"json_content": "...",
"json_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/search_9bi0sbj9xa.json",
"size_exceeded": false,
"credits_consumed": 10,
"links": [
{
"url": "https://www.bbc.com/news/articles/c3w3e467ewqo",
"title": "OpenAI to shut down Sora video platform",
"description": "OpenAI says it will discontinue its Sora app...",
"markdown_content": "# OpenAI to shut down Sora video platform\n\nOpenAI says it will discontinue..."
},
{
"url": "https://www.reddit.com/r/OutOfTheLoop/comments/1s2u847/whats_going_on_with_openais_sora_shutting_down/",
"title": "What's going on with OpenAI's Sora shutting down?",
"description": "Reddit thread discussing the shutdown.",
"markdown_content": "# What's going on with OpenAI's Sora shutting down?\n\n*r/OutOfTheLoop · u/rm-minus-r · 1mo ago*\n\n..."
}
]
}
}
```
Each link in `result.links` contains:
| Field | Type | Description |
| ------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `url` | string | The URL of the search result. |
| `title` | string | The title of the result page. |
| `description` | string | A short snippet describing the result. |
| `markdown_content` | string | Markdown content of the page. Only present when `scrape_options.formats` includes `"markdown"`. `null` if the scrape failed, was empty, or hit the global timeout. |
| `html_content` | string | HTML content of the page. Only present when `scrape_options.formats` includes `"html"`. `null` on failure/timeout. |
The full result is also available as a hosted JSON file at `result.json_hosted_url` — useful when `result.size_exceeded` is `true`.
## Retrieving a past search
`GET /v1/searches/{search_id}` returns whatever was persisted at search time, including any scraped content. It's a pure idempotent read — no re-scraping, no re-billing. Older searches without `scrape_options` simply have no per-link content fields.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
search = client.searches.get(search_id="search_9bi0sbj9xa")
print(search.id, len(search.links))
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const search = await client.searches.get('search_9bi0sbj9xa')
console.log(search.id, search.links.length)
```
```bash cURL theme={null}
curl -s "https://api.olostep.com/v1/searches/search_9bi0sbj9xa" \
-H "Authorization: Bearer $OLOSTEP_API_KEY"
```
See [Get Search](/api-reference/searches/get) for full details.
## Pricing
Each search costs **5 credits** for the search itself.
When `scrape_options` is provided, each scraped page is billed at the standard `/v1/scrapes` rate (typically 1 credit per page; some parsers cost more). The total is returned in `credits_consumed`.
Examples:
| Request | `credits_consumed` |
| ---------------------------------------- | ------------------ |
| Search only | `5` |
| Search + 5 scraped pages (1 credit each) | `10` |
# Skills
Source: https://docs.olostep.com/features/skills
Install and manage Olostep skills in your AI coding agents
Install Olostep skill folders directly into your AI coding agents from the command line.
* Skills land in a canonical store (`~/.agents/skills`) and are symlinked or copied into each agent's skills directory
* Tracks all installs in a lockfile at `~/.agents/.skill-lock.json`
* Works with Cursor, Claude, Codex, Windsurf, Continue, and more
This feature is available via the [Olostep CLI](/sdks/cli).
## The skills
`olostep add skills` installs all 13. Each is a `SKILL.md` your agent reads to know **what** Olostep can do and **when** to use it.
Skills pair with the [MCP server](/integrations/mcp-server): the MCP server gives the agent the live *tools*, the skills give it the *know-how* for when and how to use them.
### Setup
Start here — teaches the agent how to configure the Olostep MCP server.
| Skill | What it does |
| ------- | --------------------------------------------------------- |
| `setup` | Configure the Olostep MCP server so all other skills work |
### Core web data
The main Olostep capabilities — use these to give your agent live web access.
| Skill | What it does |
| ---------------- | ------------------------------------------------------------- |
| `scrape` | Turn one URL into clean markdown / HTML / JSON / text |
| `search` | Live web search — results, answers, and in-site URL discovery |
| `answers` | Cited, structured answers from live web data |
| `crawl` | Autonomously crawl a whole site |
| `map` | Discover every URL on a site |
| `batch` | Scrape up to 10,000 URLs in parallel |
| `extract-schema` | Scrape a page into structured JSON matching a schema |
### Build & integrate
For agents helping developers add Olostep to a codebase or keep up with API changes.
| Skill | What it does |
| -------------- | ------------------------------------------------ |
| `integrate` | Auto-install the Olostep SDK into a project |
| `docs-to-code` | Scrape API docs and write working code from them |
| `migrate-code` | Read a migration guide and update local code |
### Research & debug
For agents doing research, making decisions, or debugging code errors.
| Skill | What it does |
| ------------- | -------------------------------------------------------- |
| `research` | Cited, comparative web research for a decision |
| `debug-error` | Look up an error message against live GitHub / SO / docs |
### Categories
Every skill belongs to one of three categories, which map to the `--category` flag:
* **`usage`** — use Olostep's features: `scrape`, `search`, `answers`, `crawl`, `map`, `batch`, `extract-schema`.
* **`build`** — install and integrate Olostep into a codebase: `setup` (configure the MCP server) and `integrate` (add the Olostep SDK to a project).
* **`workflow`** — produce a deliverable with Olostep: `research`, `docs-to-code`, `migrate-code`, `debug-error`.
Install just one category:
```bash theme={null}
olostep add skills --category usage # core web-data skills
olostep add skills --category build # setup + integration skills
olostep add skills --category workflow # research + deliverable skills
```
### Hosted copies
Each installed skill also references its hosted copy at `https://www.olostep.com/skills//SKILL.md` — an agent with web access can fetch the latest version (which may include capabilities added since install) and falls back to the local copy otherwise. Browse all hosted skills at [olostep.com/skills](https://www.olostep.com/skills/index.md).
## Install
```bash theme={null}
npm install -g olostep-cli
# Install all 13 skills into every detected agent
olostep add skills
# Or use the skills subcommand (alias)
olostep skills install
```
Or without installing the CLI:
```bash theme={null}
npx -y olostep-cli@latest add skills
```
## Common usage
```bash theme={null}
# Install all skills
olostep skills install
# Refresh / update to latest
olostep skills update
# See what's installed and where
olostep skills list
# Remove all skills
olostep skills uninstall
```
Filter what gets installed:
```bash theme={null}
# Only core web-data skills
olostep add skills --category usage
# Only setup + integration skills
olostep add skills --category build
# Cherry-pick
olostep add skills --skill scrape --skill search --skill setup
# Specific agents only
olostep add skills --agent cursor --agent claude
# Machine-readable output
olostep add skills --json
```
## Options reference
| Option | Default | Description |
| ----------------------------------- | ------------------ | ---------------------------------------------------------- |
| `--login` | — | Run browser login before installing |
| `--source ` | CLI bundled skills | Skills source directory |
| `--cli-local-dir ` | CLI/skills | Directory where source skills are synced for CLI-local use |
| `--agent ` | — | Target a specific agent — repeatable |
| `--all-agents` / `--no-all-agents` | `--all-agents` | Target all detected agents |
| `--global` / `--no-global` | `--global` | Install into global agent skill dirs |
| `--canonical-dir ` | `~/.agents/skills` | Canonical storage location |
| `--agent-skills-dir ` | — | Custom target dir (requires `--no-global`) |
| `--skill ` | — | Include only this skill — repeatable |
| `--exclude ` | — | Exclude this skill — repeatable |
| `--category` | — | `usage`, `build`, or `workflow` |
| `--overwrite` / `--no-overwrite` | `--overwrite` | Replace existing installs |
| `--link-mode ` | `auto` | `auto` tries symlink first, falls back to copy |
| `--json` | — | Machine-readable JSON output |
**Validation rules:**
* `--link-mode` must be `auto`, `symlink`, or `copy`
* `--agent-skills-dir` requires `--no-global`
* `--no-global` requires `--agent-skills-dir`
* Unknown agent names will error
* An empty skill selection after `--skill` / `--exclude` will error
### JSON output shape
When `--json` is passed, the output looks like:
```json theme={null}
{
"sync": {
"plugin_source_dir": "/path/to/CLI/skills",
"cli_local_dir": "/path/to/CLI/skills"
},
"selected_skills": ["scrape", "search"],
"canonical_dir": "~/.agents/skills",
"lockfile_path": "~/.agents/.skill-lock.json",
"installed": [
{
"skill": "scrape",
"canonical_path": "~/.agents/skills/olostep-scrape",
"targets": [
{ "agent": "cursor", "mode": "symlink", "path": "~/.cursor/skills/olostep-scrape" },
{ "agent": "claude", "mode": "symlink", "path": "~/.claude/skills/olostep-scrape" }
]
}
]
}
```
## Supported agents
| Agent | Key |
| -------- | ---------- |
| Cursor | `cursor` |
| Claude | `claude` |
| Codex | `codex` |
| Windsurf | `windsurf` |
| Continue | `continue` |
| Augment | `augment` |
| Roo | `roo` |
| Gemini | `gemini` |
| Copilot | `copilot` |
| Factory | `factory` |
Use `--all-agents` (the default) to target all detected agents, or `--agent ` for specific ones.
## `olostep list skills`
See which skills are installed and into which agents — without digging through files.
```bash theme={null}
olostep list skills # human-readable summary
olostep list skills --json # machine-readable
```
## `olostep remove skills`
Removes Olostep-installed skill folders from the canonical store and agent skill directories, and cleans up the lockfile.
```bash theme={null}
# Remove all Olostep skills from all agents
olostep remove skills
# Remove a specific skill
olostep remove skills --skill research
# Remove from a specific agent only
olostep remove skills --agent cursor
# Machine-readable JSON output
olostep remove skills --json
```
| Option | Default | Description |
| ---------------------------------- | ------------------ | ------------------------------------------------ |
| `--agent ` | — | Remove only from specified agent(s) — repeatable |
| `--all-agents` / `--no-all-agents` | `--all-agents` | Target all detected agents |
| `--canonical-dir ` | `~/.agents/skills` | Canonical skills directory to remove from |
| `--agent-skills-dir ` | — | Custom target skills directory for removal |
| `--skill ` | — | Remove only matching skill(s) — repeatable |
| `--json` | — | Machine-readable JSON output |
**Safety:** Only folders with the `olostep-` prefix are touched. Non-Olostep skill folders in agent directories are never modified. Lockfile cleanup only removes `olostep-`-prefixed keys.
## Skill discovery and format
Skills are discovered by scanning subdirectories for a `SKILL.md` file. The file must include a YAML frontmatter block with `name` and `description` fields. Duplicate sanitized names are rejected.
```markdown theme={null}
---
name: my-skill
description: What this skill does and when to use it
---
# My Skill
...skill instructions here...
```
## Naming convention
Installed folders always use the `olostep-` prefix — e.g. `research` becomes `olostep-research`. Names are sanitized to lowercase with invalid characters replaced by `-`. This prefix keeps Olostep-managed skills clearly identifiable and prevents conflicts with other tools.
## Related
* [CLI reference](/sdks/cli) — full command documentation
* [MCP Server](/integrations/mcp-server) — give your agent live Olostep tools
# Introduction
Source: https://docs.olostep.com/features/structured-content/intro
By default, Olostep supports returning content in various formats including:
* HTML
* Text
* Raw PDF
* Markdown
**For some use cases you might not need the entire content but only specified, structured and clean data in JSON format.**
To achieve this, you may use:
* **[Parsers](./parsers)**: Ideal for high-volume, consistent, recurring website scraping
* **[LLM Extraction](./llm-extraction)**: Ideal for flexible extraction needs or websites with changing structures.
Both methods provide clean, structured JSON data that can be immediately used in your applications without additional processing.
# Using LLM Extraction
Source: https://docs.olostep.com/features/structured-content/llm-extraction
For websites with changing structures or one-off extraction needs, Olostep offers LLM-powered extraction. This approach:
* Feeds the content to a Large Language Model
* Instructs the model to parse and return only the specified data
* Returns a clean JSON structure containing exactly what you need
```python theme={null}
import requests
import json
def extract_with_llm():
url = "https://api.olostep.com/v1/scrapes"
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
data = {
"url_to_scrape": "https://www.berklee.edu/events/stefano-marchese-friends",
"formats": [
"markdown",
"json"
],
"llm_extract": {
"schema": {
"event": {
"type": "object",
"properties": {
"title": {"type": "string"},
"date": {"type": "string"},
"description": {"type": "string"},
"venue": {"type": "string"},
"address": {"type": "string"},
"start_time": {"type": "string"}
}
}
}
},
}
response = requests.post(url, headers=headers, json=data)
result = response.json()
# The LLM extract will be available in the result
print(json.dumps(result, indent=2))
return result
if __name__ == "__main__":
extract_with_llm()
```
You can either pass the `schema` or a `prompt` to the LLM.
```python theme={null}
import requests
import json
def extract_with_llm():
url = "https://api.olostep.com/v1/scrapes"
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
data = {
"url_to_scrape": "https://www.berklee.edu/events/stefano-marchese-friends",
"formats": [
"markdown",
"json"
],
"llm_extract": {
"prompt": "Extract the event title, date, description, venue, address, and start time from the event page."
},
}
response = requests.post(url, headers=headers, json=data)
result = response.json()
# The LLM extract will be available in the result
print(json.dumps(result, indent=2))
return result
if __name__ == "__main__":
extract_with_llm()
```
The `prompt` is a natural language string that is passed to the LLM to extract the data. The LLM decides how to extract the data based on the prompt. You can use this when you don't want to use a `schema`.
Sample Response:
```json theme={null}
{
"id": "scrape_94iqy385ty",
...
"result": {
"json_content": "{\"event\":{\"title\":\"Stefano Marchese and Friends\",\"date\":\"Wednesday / January 22, 2025\",\"description\":\"Join acclaimed Italian singer-songwriter and educator Stefano Marchese for an unforgettable evening of musical magic as he takes the stage alongside a constellation of extraordinary talent in a concert titled Concerto di Duetti.\",\"venue\":\"David Friend Recital Hall (DFRH)\",\"address\":\"921 Boylston Street Boston MA 02115 United States\",\"start_time\":\"7:30 p.m. (EST)\"}}"
}
}
```
json\_content is the stringified JSON content of the event. You can access it as a JSON object by parsing the string.
```python theme={null}
import json
event = json.loads(result["json_content"])
print(event["event"]["title"])
```
# Using Parsers
Source: https://docs.olostep.com/features/structured-content/parsers
Parsers are a way to turn unstructured data in structured data that is compatible with your backend. Combining parsers with the Olostep API (crawls, scrapes, batches) allows to turn any website into an API call to return the JSON you need.
Parsers are ideal when you need data at scale in a recurrent way from the same websites. This approach is significantly more cost-efficient and fast compared to [LLM extract](/features/structured-content/llm-extraction) and returns only the required JSON.
We offer pre-built parsers for popular websites and use cases. You can also create your own parsers in a few minutes with LLMs using the [dashboard](https://www.olostep.com/dashboard/parsers) or ask our team to do it for you.
### Pre-Built Parsers
We offer several pre-built parsers for popular websites:
* Google Search: `@olostep/google-search`
* Amazon Product: `@olostep/amazon-it-product`
* Extract Email: `@olostep/extract-emails`
* Extract Calendars: `@olostep/extract-calendars`
* Extract Socials: `@olostep/extract-socials`
* TikTok data extraction: get in touch with us to get the parser ID
* Google News: get in touch with us to get the parser ID
* Google Maps: get in touch with us to get the parser ID
### Example Usage
```python theme={null}
import requests
import json
endpoint = "https://api.olostep.com/v1/scrapes"
payload = {
"formats": ["json"],
"parser": {"id": "@olostep/google-search"},
"url_to_scrape": "https://www.google.com/search?q=alexander+the+great&gl=us&hl=en",
"wait_before_scraping": 0,
}
headers = {
"Authorization": "Bearer ",
"Content-Type": "application/json"
}
response = requests.request("POST", endpoint, json=payload, headers=headers)
print(json.dumps(response.json(), indent=4))
```
## Response Format
When you make a request to the Olostep API with the parser format, you'll receive a JSON response like the example below:
```json theme={null}
{
"id": "scrape_94iqy385ty",
"object": "scrape",
"created": 1740595134,
"metadata": {},
"retrieve_id": "94iqy385ty",
"url_to_scrape": "https://www.google.com/search?q=alexander+the+great&gl=us&hl=en",
"result": {
"html_content": null,
"markdown_content": null,
"text_content": null,
"json_content": "{\"searchParameters\":{\"type\":\"search\",\"engine\":\"google\",\"q\":\"alexander the great\"},\"knowledgeGraph\":{\"title\":\"Alexander the Great\",\"type\":\"Former King of Macedonia\",\"description\":\"Alexander III of Macedon, most commonly known as Alexander the Great, was a king of the ancient Greek kingdom of Macedon.\",\"imageUrl\":\"https://www.mayaincaaztec.com/ancient-greece/alexander-the-great\",\"attributes\":{\"Born\":\"July 356 BC, Pella\",\"Died\":\"June 323 BC (age 32 years), Babylon\",\"Spouse\":\"Roxana (m. 327 BC\u2013323 BC), Parysatis II (m. 324 BC\u2013323 BC), Stateira (m. 324 BC\u2013323 BC)\",\"Children\":\"Alexander IV of Macedon\",\"Full name\":\"Alexander III of Macedon\",\"Siblings\":\"Cleopatra of Macedon, Philip III of Macedon, Thessalonike of Macedon, Cynane, Caranus, Europa of Macedon\"}},\"organic\":[{\"title\":\"Alexander the Great\",\"link\":\"https://en.wikipedia.org/wiki/Alexander_the_Great\",\"position\":1,\"snippet\":\"He was undefeated in battle and is widely considered to be one of history's greatest and most successful military commanders.\",\"sitelinks\":[{\"title\":\"Death of Alexander the Great\",\"link\":\"https://en.wikipedia.org/wiki/Death_of_Alexander_the_Great\"},{\"title\":\"Wars of Alexander the Great\",\"link\":\"https://en.wikipedia.org/wiki/Wars_of_Alexander_the_Great\"}]},{\"title\":\"Alexander the Great | Biography, Empire, Death, & Facts\",\"link\":\"https://www.britannica.com/biography/Alexander-the-Great#:~:text=Top%20Questions-,Why%20is%20Alexander%20the%20Great%20famous%3F,Greece%20to%20part%20of%20India.\",\"position\":2},{\"title\":\"Alexander the Great's Last Three Wishes. - LinkedIn\",\"link\":\"https://www.linkedin.com/pulse/moment-can-last-lifetime-alexander-greats-three-wishes-holt#:~:text=1)%20The%20king%20of%20Macedon,my%20coffin%2C%22%20Alexander%20said.\",\"position\":3},{\"title\":\"Alexander the Great Failure: The Collapse of the Macedonian Empire\",\"link\":\"https://www.publishersweekly.com/9781847251886#:~:text=His%20inability%20to%20delegate%20work,its%20independence%20and%20its%20boundaries.\",\"position\":4},{\"title\":\"Who defeated Alexander The Great? Who conquered Greece after him ...\",\"link\":\"https://www.quora.com/Who-defeated-Alexander-The-Great-Who-conquered-Greece-after-him-and-why-were-they-able-to-conquer-that-region-while-Alexander-couldnt#:~:text=No%20one%20defeated%20Alexander%20the,his%20death%20was%20not%20natural.\",\"position\":5},{\"title\":\"Alexander the Great | Biography, Empire, Death, & Facts\",\"link\":\"https://www.britannica.com/biography/Alexander-the-Great\",\"position\":6,\"snippet\":\"Feb 11, 2025 \u2014 Alexander the Great, a fearless Macedonian king and military genius, conquered vast territories from Greece to Egypt and India, ...\"},{\"title\":\"Alexander the Great: Empire & Death\",\"link\":\"https://www.history.com/topics/ancient-greece/alexander-the-great\",\"position\":7,\"snippet\":\"Nov 9, 2009 \u2014 Alexander the Great was an ancient Macedonian ruler and one of history's greatest military minds who, as King of Macedonia and Persia, ...\"},{\"title\":\"History - Alexander the Great\",\"link\":\"https://www.bbc.co.uk/history/historic_figures/alexander_the_great.shtml\",\"position\":8,\"snippet\":\"Alexander III of Macedon, better known as Alexander the Great, single-handedly changed the nature of the ancient world in little more than a decade.\"},{\"title\":\"Alexander the Great - National Geographic Education\",\"link\":\"https://education.nationalgeographic.org/resource/alexander-great/\",\"position\":9,\"snippet\":\"Oct 19, 2023 \u2014 Alexander was born in 356 B.C.E. in Pella, Macedonia, to King Philip II. As a young boy, Alexander was taught to read, write, and play the lyre.\"},{\"title\":\"Who loved Alexander the Great?\",\"link\":\"https://museums.cam.ac.uk/magic/who-loved-alexander-great\",\"position\":10,\"snippet\":\"Throughout his life, Alexander married 3 women and fathered at least 2 children but also had several male lovers. Amongst his closest relationships was that ...\"},{\"title\":\"Alexander the Great (1956)\",\"link\":\"https://www.imdb.com/title/tt0048937/\",\"position\":11,\"snippet\":\"The life and military conquests of Alexander III of Macedon (July 20/21, 356 - June 10/11, 323 B.C.), commonly known as Alexander the Great.\"},{\"title\":\"Alexander the Great\",\"link\":\"https://www.worldhistory.org/Alexander_the_Great/\",\"position\":12,\"snippet\":\"Nov 14, 2013 \u2014 He is known as 'the great' both for his military genius and his diplomatic skills in handling the various populaces of the regions he conquered.\"}],\"peopleAlsoAsk\":[{\"question\":\"What is Alexander the Great most famous for?\"},{\"question\":\"What did Alexander the Great say before he died?\",\"link\":\"https://www.linkedin.com/pulse/moment-can-last-lifetime-alexander-greats-three-wishes-holt#:~:text=1)%20The%20king%20of%20Macedon,my%20coffin%2C%22%20Alexander%20said.\",\"title\":\"Alexander the Great's Last Three Wishes. - LinkedIn\"},{\"question\":\"What led to the fall of Alexander?\",\"link\":\"https://www.publishersweekly.com/9781847251886#:~:text=His%20inability%20to%20delegate%20work,its%20independence%20and%20its%20boundaries.\",\"title\":\"Alexander the Great Failure: The Collapse of the Macedonian Empire\"},{\"question\":\"Which country defeated Alexander the Great?\",\"link\":\"https://www.quora.com/Who-defeated-Alexander-The-Great-Who-conquered-Greece-after-him-and-why-were-they-able-to-conquer-that-region-while-Alexander-couldnt#:~:text=No%20one%20defeated%20Alexander%20the,his%20death%20was%20not%20natural.\",\"title\":\"Who defeated Alexander The Great? Who conquered Greece after him ...\"}],\"relatedSearches\":[{\"query\":\"Alexander the Great book\"},{\"query\":\"Alexander the Great empire\"},{\"query\":\"Alexander the Great death\"},{\"query\":\"Alexander the Great religion\"},{\"query\":\"Alexander the Great Empire map\"},{\"query\":\"Alexander the Great achievements\"},{\"query\":\"What was Alexander the Great known for\"},{\"query\":\"Alexander the Great empire name\"}]}",
"llm_extract": null,
"screenshot_hosted_url": null,
"html_hosted_url": null,
"markdown_hosted_url": null,
"json_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json_94iqy385ty.json",
"text_hosted_url": null,
"links_on_page": [],
"page_metadata": {
"status_code": 200,
"title": ""
}
}
}
```
The response contains:
* **Basic request information**: `id`, `object`, `created` timestamp, `url_to_scrape`
* **Result object** with URLs to access different formats of the data
* **json\_content** with structured JSON results including:
* `searchParameters`: Information about the search query
* `knowledgeGraph`: Detailed information about the search subject (when available)
* `organic`: List of search results with title, link, position, and snippet
* `peopleAlsoAsk`: Related questions that users commonly search for
* `relatedSearches`: Suggested related search queries
`json_content` is the main part of the response with the structured JSON results. You can access the JSON content directly from the response or use the hosted URL provided in the response.
## Structured Response: json\_content
```json theme={null}
{
"searchParameters": {
"type": "search",
"engine": "google",
"q": "alexander the great"
},
"knowledgeGraph": {
"title": "Alexander the Great",
"type": "Former King of Macedonia",
"description": "Alexander III of Macedon, most commonly known as Alexander the Great, was a king of the ancient Greek kingdom of Macedon.",
"imageUrl": "https://www.mayaincaaztec.com/ancient-greece/alexander-the-great",
"attributes": {
"Born": "July 356 BC, Pella",
"Died": "June 323 BC (age 32 years), Babylon",
"Spouse": "Roxana (m. 327 BC–323 BC), Parysatis II (m. 324 BC–323 BC), Stateira (m. 324 BC–323 BC)",
"Children": "Alexander IV of Macedon",
"Full name": "Alexander III of Macedon",
"Siblings": "Cleopatra of Macedon, Philip III of Macedon, Thessalonike of Macedon, Cynane, Caranus, Europa of Macedon"
}
},
"organic": [
{
"title": "Alexander the Great",
"link": "https://en.wikipedia.org/wiki/Alexander_the_Great",
"position": 1,
"snippet": "He was undefeated in battle and is widely considered to be one of history's greatest and most successful military commanders.",
"sitelinks": [
{
"title": "Death of Alexander the Great",
"link": "https://en.wikipedia.org/wiki/Death_of_Alexander_the_Great"
},
{
"title": "Wars of Alexander the Great",
"link": "https://en.wikipedia.org/wiki/Wars_of_Alexander_the_Great"
}
]
},
{
"title": "Alexander the Great Failure: The Collapse of the Macedonian Empire",
"link": "https://www.publishersweekly.com/9781847251886#:~:text=His%20inability%20to%20delegate%20work,its%20independence%20and%20its%20boundaries.",
"position": 2
},
{
"title": "Which Indian king first time defeated Alexander? - Quora",
"link": "https://www.quora.com/Which-Indian-king-first-time-defeated-Alexander#:~:text=Alexander%20the%20Great%20was%20defeated,returned%20with%20only%2020000%20troops.",
"position": 3
},
{
"title": "Alexander the Great | Biography, Empire, Death, & Facts",
"link": "https://www.britannica.com/biography/Alexander-the-Great",
"position": 4,
"snippet": "Feb 11, 2025 — Alexander the Great, a fearless Macedonian king and military genius, conquered vast territories from Greece to Egypt and India, ..."
},
{
"title": "Alexander the Great: Empire & Death",
"link": "https://www.history.com/topics/ancient-greece/alexander-the-great",
"position": 5,
"snippet": "Nov 9, 2009 — Alexander the Great was an ancient Macedonian ruler and one of history's greatest military minds who, as King of Macedonia and Persia, ..."
},
{
"title": "Alexander the Great (1956)",
"link": "https://www.imdb.com/title/tt0048937/",
"position": 6,
"snippet": "The life and military conquests of Alexander III of Macedon (July 20/21, 356 - June 10/11, 323 B.C.), commonly known as Alexander the Great."
},
{
"title": "History - Alexander the Great",
"link": "https://www.bbc.co.uk/history/historic_figures/alexander_the_great.shtml",
"position": 7,
"snippet": "Alexander III of Macedon, better known as Alexander the Great, single-handedly changed the nature of the ancient world in little more than a decade."
},
{
"title": "Who loved Alexander the Great?",
"link": "https://museums.cam.ac.uk/magic/who-loved-alexander-great",
"position": 8,
"snippet": "Throughout his life, Alexander married 3 women and fathered at least 2 children but also had several male lovers. Amongst his closest relationships was that ..."
},
{
"title": "Alexander the Great",
"link": "https://www.worldhistory.org/Alexander_the_Great/",
"position": 9,
"snippet": "Nov 14, 2013 — He is known as 'the great' both for his military genius and his diplomatic skills in handling the various populaces of the regions he conquered."
},
{
"title": "Alexander the Great - National Geographic Education",
"link": "https://education.nationalgeographic.org/resource/alexander-great/",
"position": 10,
"snippet": "Oct 19, 2023 — Alexander was born in 356 B.C.E. in Pella, Macedonia, to King Philip II. As a young boy, Alexander was taught to read, write, and play the lyre."
}
],
"peopleAlsoAsk": [
{
"question": "What is Alexander the Great most famous for?"
},
{
"question": "What did Alexander the Great say before he died?"
},
{
"question": "What led to the fall of Alexander?",
"link": "https://www.publishersweekly.com/9781847251886#:~:text=His%20inability%20to%20delegate%20work,its%20independence%20and%20its%20boundaries.",
"title": "Alexander the Great Failure: The Collapse of the Macedonian Empire"
},
{
"question": "Who first defeated Alexander the Great?",
"link": "https://www.quora.com/Which-Indian-king-first-time-defeated-Alexander#:~:text=Alexander%20the%20Great%20was%20defeated,returned%20with%20only%2020000%20troops.",
"title": "Which Indian king first time defeated Alexander? - Quora"
}
],
"relatedSearches": [
{
"query": "Alexander the Great book"
},
{
"query": "Alexander the Great empire"
},
{
"query": "Alexander the Great death"
},
{
"query": "Alexander the Great religion"
},
{
"query": "Alexander the Great Empire map"
},
{
"query": "Alexander the Great achievements"
},
{
"query": "What was Alexander the Great known for"
},
{
"query": "Alexander the Great movie"
}
]
}
```
Olostep provides also a hosted JSON file with the structured results. You can access the JSON file using the `json_hosted_url` field in the response:
* Structured JSON: [View example JSON](https://olostep-storage.s3.us-east-1.amazonaws.com/json_vxc86vq2pf.json)
If you want to also get the HTML and Markdown content of the search results, you can include these formats in the `formats` parameter and Olostep will return them in the response and provide hosted URLs for each format.
* [Markdown format](https://olostep-storage.s3.us-east-1.amazonaws.com/markDown_vxc86vq2pf.txt)
* [HTML](https://olostep-storage.s3.us-east-1.amazonaws.com/text_vxc86vq2pf.txt)
### Need a Custom Parser?
If you need a parser for a specific website or the ID of a pre-built parser:
* Contact us at [info@olostep.com](mailto:info@olostep.com)
# Authentication
Source: https://docs.olostep.com/get-started/authentication
Authenticate to the Olostep search, scraping, and crawling API with API keys.
The API endpoints require that you authenticate using an API token.
## Generate a token
The token can be generated from the Olostep dashboard. Please create an account [here](https://www.olostep.com/auth/).
## Use your token
You can authenticate by adding an `Authorization` header to all your HTTP calls. The Authorization header is formatted as such: `Authorization: Bearer ` (replace `` with your token. If you don't have a token, you can generate one for free from the Olostep [dashboard](https://www.olostep.com/dashboard/).
Examples:
```python Python theme={null}
# pip install requests
import requests
endpoint = 'https://api.olostep.com/v1/scrapes/'
headers = {
'Authorization': 'Bearer ',
'Accept': 'application/json'
}
response = requests.get(endpoint, headers=headers)
print(response.status_code)
print(response.json())
```
```js Node theme={null}
// npm install node-fetch
// ESM
import fetch from 'node-fetch'
const endpoint = 'https://api.olostep.com/v1/scrapes/'
const res = await fetch(endpoint, {
headers: {
'Authorization': 'Bearer ',
'Accept': 'application/json'
}
})
console.log(res.status)
console.log(await res.json())
```
```bash cURL theme={null}
curl -L -X GET 'https://api.olostep.com/v1/scrapes/' \
-H 'Accept: application/json' \
-H 'Authorization: Bearer '
```
## Using the CLI
If you're using the [Olostep CLI](/sdks/cli), you can skip manual key handling and sign in from the terminal:
```bash theme={null}
npm install -g olostep-cli
olostep login
```
The browser opens, you click **Authorize**, and the key is saved locally. For CI, set `OLOSTEP_API_KEY` instead.
## Using the SDKs
The [Python](/sdks/python) and [Node.js](/sdks/node-js) SDKs read `OLOSTEP_API_KEY` from the environment, or accept the key directly when you construct the client.
# Welcome to Olostep
Source: https://docs.olostep.com/get-started/welcome
Olostep: Infrastructure for the Web's second user. The best search, scraping and crawling API for AI.
Olostep is infrastructure for the Web's second user — giving AI agents a way to search the web, extract structured data in real time, and build custom research agents.
![Main dashboard interface]()
## Introduction
* The **Olostep API** is the best **web search**, **scraping** and **crawling** API for AI used by some of the leading startups and scaleups in the world.
* The **Olostep Agent** allows to automate **research workflows** in a no code way with just a prompt in natural language.
**For AI agents:** fetch [docs.olostep.com/llms.txt](https://docs.olostep.com/llms.txt) for a complete index of this documentation before exploring further or [get started here](https://www.olostep.com/agent-onboarding/SKILL.md).
## Use Olostep from your terminal and AI agents
Beyond the API, Olostep ships a CLI, an MCP server, and drop-in skills so any tool — Claude Code, Cursor, Windsurf, and more — can use the web natively.
`npm i -g olostep-cli` — scrape, map, crawl, answer, and batch the web from your terminal. JSON output for scripts, CI, and agents.
Give any MCP client (Claude, Cursor, VS Code) live web tools. Hosted endpoint — no install.
Drop-in skills that teach AI coding agents how and when to use Olostep. Install with `olostep add skills`.
## What can Olostep do?
Pull any URL as clean Markdown, HTML, screenshots, or structured JSON.
Recursively gather every page on a site, with filters and search.
Get AI-synthesised answers from live web sources, with citations.
### Why Olostep?
* **Built for AI**: Clean Markdown, structured JSON, citations — output your agents and apps consume directly.
* **Reliable at scale**: Industry-leading success rate; handles JavaScript, anti-bot, and proxies under the hood.
* **Fast**: Sub-second single scrape; up to 10,000 URLs in a single batch in 5–7 minutes.
* **Cost-effective**: Significantly cheaper than alternatives at production scale.
* **CLI + MCP + Skills**: Use Olostep from your terminal, scripts, or any MCP-aware agent — agent skills included.
***
## Scrape
Pull any URL as clean Markdown. See the [Scrape feature docs](/features/scrapes) for all options.
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
result = client.scrapes.create(
url_to_scrape="https://en.wikipedia.org/wiki/Alexander_the_Great",
formats=["markdown"],
)
print(result.markdown_content)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const result = await client.scrapes.create({
url: 'https://en.wikipedia.org/wiki/Alexander_the_Great',
formats: ['markdown'],
})
console.log(result.markdown_content)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/scrapes" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url_to_scrape": "https://en.wikipedia.org/wiki/Alexander_the_Great",
"formats": ["markdown"]
}'
```
```bash CLI theme={null}
olostep scrape "https://en.wikipedia.org/wiki/Alexander_the_Great"
```
```json theme={null}
{
"id": "scrape_6h89o8u1kt",
"object": "scrape",
"result": {
"markdown_content": "## Alexander the Great...",
"markdown_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/markDown_6h89o8u1kt.txt",
"page_metadata": { "status_code": 200, "title": "Alexander the Great - Wikipedia" }
}
}
```
## Crawl
Recursively gather every page on a site, with include/exclude filters and an optional `search_query` to focus the crawl. See [Crawl feature docs](/features/crawls).
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
crawl = client.crawls.create(
start_url="https://docs.olostep.com",
max_pages=50,
)
for page in crawl.pages():
print(page.url)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const crawl = await client.crawls.create({
url: 'https://docs.olostep.com',
maxPages: 50,
})
for await (const page of crawl.pages()) {
console.log(page.url)
}
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/crawls" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"start_url": "https://docs.olostep.com",
"max_pages": 50
}'
```
```bash CLI theme={null}
olostep crawl "https://docs.olostep.com" --max-pages 50
```
```json theme={null}
{
"id": "crawl_abc123",
"object": "crawl",
"status": "completed",
"pages_count": 47,
"pages": [
{ "url": "https://docs.olostep.com/get-started/welcome", "retrieve_id": "..." },
{ "url": "https://docs.olostep.com/features/scrapes", "retrieve_id": "..." }
]
}
```
## Answer
Ask a question and get an AI-synthesised answer from live web sources, with citations. Pass a JSON schema to shape the output. See [Answers docs](/features/answers).
```python Python theme={null}
from olostep import Olostep
client = Olostep(api_key="YOUR_REAL_KEY")
answer = client.answers.create(task="What does Olostep do?")
print(answer.result)
print(answer.sources)
```
```js Node theme={null}
import Olostep from 'olostep'
const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
const answer = await client.answers.create({ task: 'What does Olostep do?' })
console.log(answer.result)
console.log(answer.sources)
```
```bash cURL theme={null}
curl -s -X POST "https://api.olostep.com/v1/answers" \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"task": "What does Olostep do?"}'
```
```bash CLI theme={null}
olostep answer "What does Olostep do?"
```
```json theme={null}
{
"id": "answer_abc123",
"object": "answer",
"task": "What does Olostep do?",
"result": {
"json_content": "{\"result\":\"Olostep is an API that lets AI agents search, scrape, and structure web data.\"}",
"sources": [
"https://docs.olostep.com/get-started/welcome",
"https://www.olostep.com/"
]
}
}
```
***
## More capabilities
Scrape up to 10,000 URLs in parallel; results back in 5–7 minutes.
Discover every URL on a site with include/exclude patterns.
Live web search with structured links and optional inline scraping.
Self-healing extractors that turn pages into typed JSON at scale.
Run scrapes, crawls, and answers on a recurring schedule.
Upload files for batches or to connect your knowledge base.
***
## Resources
Check out all supported features for your scraping and AI search needs.
Start using the API and test out the various params.
Use Olostep in n8n, make, relay, zapier, etc
Browse ready-to-use examples to get started quickly.
# Olostep + Apify Integration
Source: https://docs.olostep.com/integrations/apify
Automate web search, scraping and crawling with Apify Actors using Olostep — the API to search, extract and structure web data.
Olostep is a Web search, scraping and crawling API — an API to search, extract and structure web data. This guide shows how to use Olostep with Apify Actors to build reliable web data pipelines end‑to‑end.
## What you can build
Extract content from any single URL in Markdown, HTML, JSON, or Text
Process large lists of URLs in parallel with structured outputs
Discover and scrape linked pages to build complete datasets
Extract all URLs from a website (sitemap-like discovery)
Ask questions and get structured JSON answers with sources
## Quick start
### 1) Install Apify CLI
```bash theme={null}
npm install -g apify-cli
apify --version
```
### 2) Get your Olostep API key
From the Olostep Dashboard → API Keys.
### 3) Run the Olostep Actor locally
```bash theme={null}
cd olostep-tools/integrations/apify
apify run
```
Default local input file lives at:
`olostep-tools/integrations/apify/storage/key_value_stores/default/INPUT.json`
Example input:
```json theme={null}
{
"operation": "scrape",
"apiKey": "YOUR_OLostep_API_KEY",
"url_to_scrape": "https://example.com",
"formats": "markdown"
}
```
### 4) Deploy to Apify (cloud)
```bash theme={null}
apify login
apify push
```
Then open Apify Console → Actors → run the actor with your desired input.
### Run in Apify Console (step by step)
1. Open your Actor in Apify Console → Source → Input.
2. In the Manual tab you’ll see a visible “Olostep API Key” field. Paste your key from the Olostep Dashboard.
3. Choose an operation (defaults to “scrape”).
4. Fill the relevant fields (for “scrape”, set “URL to Scrape”).
5. Click Save → Start.
6. When the run finishes, open the Dataset tab to download results (JSON/CSV/Excel).
Notes:
* For “URL to Scrape”, you can paste with or without scheme. If missing, the actor automatically prepends `https://`.
* If a site is heavy in JavaScript and you see a timeout, set “Wait Before Scraping” to 2000–5000 ms and run again.
## Available operations
### Scrape Website
Extract content from a single URL. Great for page‑level automation.
Must be "scrape"
Your Olostep API key (Bearer)
The URL to scrape (must include http\:// or https\://)
One of: Markdown, HTML, JSON, Text
Optional country code (e.g., "US", "GB", "CA")
Optional wait time in ms for JavaScript rendering (0–10000)
Optional parser ID (e.g., "@olostep/amazon-product")
Output fields:
* id, url, status, formats
* markdown\_content / html\_content / json\_content / text\_content
* hosted URLs (if available), page metadata
### Batch Scrape URLs
Process many URLs at once with consistent formatting and structure.
Must be "batch"
Your Olostep API key
JSON array of objects with `url` and optional `custom_id`\
Example: `[{"url":"https://example.com","custom_id":"site1"}]`
One of: Markdown, HTML, JSON, Text
Optional country code
Optional wait time in ms for JS sites
Optional parser ID
Output fields:
* batch\_id, status, total\_urls, created\_at, formats, country, parser, urls\[]
### Create Crawl
Follow links and scrape multiple pages from a start URL.
Must be "crawl"
Your Olostep API key
Starting URL for the crawl
Max pages to crawl. Set to `1` to scrape only the start URL.
One of: Markdown, HTML, JSON, Text
Optional country code
Optional parser ID
Output fields:
* crawl\_id, object, status, start\_url, max\_pages, created, formats
### Create Map
Discover all URLs on a website and prepare for later batch scraping.
Must be "map"
Your Olostep API key
The website to map
Optional query filter
Limit number of URLs
Include glob(s), e.g. "/products/\*\*"
Exclude glob(s), e.g. "/admin/\*\*"
Output fields:
* map\_id, object, website\_url, total\_urls, urls\[], search\_query, top\_n
## Copy‑paste JSON examples (Console → Input → JSON)
### Scrape
```json theme={null}
{
"operation": "scrape",
"apiKey": "YOUR_OLOSTEP_API_KEY",
"url_to_scrape": "https://www.wikipedia.org",
"formats": "markdown",
"wait_before_scraping": 2000
}
```
### Batch
```json theme={null}
{
"operation": "batch",
"apiKey": "YOUR_OLOSTEP_API_KEY",
"batch_array": "[{\"url\":\"https://example.com\",\"custom_id\":\"site1\"},{\"url\":\"https://olostep.com\",\"custom_id\":\"site2\"}]",
"formats": "json"
}
```
### Crawl
```json theme={null}
{
"operation": "crawl",
"apiKey": "YOUR_OLOSTEP_API_KEY",
"start_url": "https://docs.example.com",
"max_pages": 50,
"formats": "markdown"
}
```
### Map
```json theme={null}
{
"operation": "map",
"apiKey": "YOUR_OLOSTEP_API_KEY",
"website_url": "https://example.com",
"include_patterns": "/blog/**",
"top_n": 200
}
```
### Answers
```json theme={null}
{
"operation": "answers",
"apiKey": "YOUR_OLOSTEP_API_KEY",
"task": "What is the latest funding round of Olostep? Provide company, round, date, amount.",
"json": "{\"company\":\"\",\"round\":\"\",\"date\":\"\",\"amount\":\"\"}"
}
```
## Example workflows
1. Create Map → include "/products/\*\*"
2. Parse URLs → build batch array
3. Batch Scrape URLs → formats: JSON
4. Send to Google Sheets / Airtable
1. Schedule actor (daily)
2. Scrape Website → formats: Markdown
3. Summarize with LLM
4. Notify on Slack
1. Create Crawl (blog/docs)
2. Store outputs in Notion
3. Refresh weekly with Schedule
## Specialized parsers
Olostep supports parsers to structure data for popular sites.
`@olostep/amazon-product` → title, price, rating, reviews, images, variants
`@olostep/google-search` → results, titles, snippets, URLs
`@olostep/google-maps` → business info, reviews, ratings, location
Explore email extractors, social handle finders, calendar link extractors, and more
## Best practices
Faster, cheaper, easier to monitor and respect rate limits.
JS‑heavy sites: increase `wait_before_scraping` (e.g., 2000–5000ms).
Avoid unnecessary tasks — check changes first, keep deduplication state.
Use hosted outputs to bypass payload size limits in Apify flows.
Batch/Crawl/Map return IDs; retrieve later or chain with a delay.
If you see a 504 or transient timeout, the actor automatically retries once with a short wait time.\
You can also set “Wait Before Scraping” to 2000–5000 ms for JS‑heavy pages.
## Troubleshooting
* Check API key from dashboard
* Remove trailing spaces
* Re‑enter in Apify input form
* Increase wait time
* Verify URL is public / not login‑gated
* Try different output format
* Space runs via schedule
* Prefer batch for many URLs
* Upgrade Olostep plan if needed
* Try country parameter
* Adjust wait and parser
* Contact support for guidance
## Pricing
Olostep charges by API usage (independent of Apify):
* Scrapes → per scrape
* Batches → per URL
* Crawls → per page
* Maps → per operation
See `https://www.olostep.com/pricing`.
## Security
* Your API key is sent as Bearer token at runtime.
* Do not commit keys to version control; Apify stores inputs in Key‑Value Store.
* In local development, keep keys in `storage/key_value_stores/default/INPUT.json` (gitignored).
## Related resources
Extract LLM‑friendly Markdown, HTML, text or structured JSON from any URL.
Process up to 10k URLs concurrently and retrieve results later.
Recursively discover and scrape a site’s content.
Get all URLs on a website to prepare batch scrapes.
## Support
Apify platform
Apify platform & SDK docs
Complete API docs
[info@olostep.com](mailto:info@olostep.com)
# Olostep + OpenClaw Integration
Source: https://docs.olostep.com/integrations/clawhub
Give your OpenClaw agent live web access with 13 skills and a 9-tool MCP server
Your OpenClaw agent can read documentation, but it can't read the web. The [Olostep Web Agent plugin](https://clawhub.ai/plugins/olostep-web-agent) fixes that — search, scrape, crawl, and extract structured data from any website, directly inside your agent workflow.
One install gives you **13 skills** for high-level tasks (debug an error from live StackOverflow threads, write integration code from current docs, research tools with structured comparisons) and a **9-tool MCP server** for direct programmatic access. JS-heavy SPAs, CAPTCHAs, Cloudflare, residential proxies — handled automatically.
## Installation
```bash theme={null}
clawhub install olostep
```
That's it. If you prefer to wire the MCP server manually, add this to your OpenClaw configuration:
```json theme={null}
{
"mcpServers": {
"olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "your-api-key-here"
}
}
}
}
```
Get a free API key at [olostep.com/auth](https://olostep.com/auth) — 500 requests/month, no credit card.
## Skills
### Core Data Skills
These six skills are the building blocks. Each one does one thing well, and your agent can compose them for complex workflows.
Any URL to clean markdown, HTML, JSON, or text. Full browser rendering, anti-bot bypass, geo-targeting, browser actions (click, scroll, type), and pre-built parsers for common site types.
Three modes: AI-synthesized answers with citations, raw Google SERP data (organic results, PAA, knowledge graph), and domain-scoped URL discovery.
Start from one URL, follow links, scrape every page discovered. Set max pages, include/exclude URL patterns, and control crawl depth.
Scrape up to 10,000 URLs in parallel with full rendering. Tag each URL with `custom_id` to map results back to your sources.
Discover every URL on a website without scraping any of them. Filter by glob patterns, rank by search query relevance, cap with `top_n`.
Ask a plain-language question, get an AI-synthesized answer grounded in live web sources. Pass a `json` parameter to get structured output matching any schema you define.
### Workflow Skills
These seven skills chain multiple core operations together. They handle the thinking — your agent just picks the right skill for the job.
"Compare the top 3 ORMs for Node.js" — the agent searches multiple sources, scrapes pricing and feature pages, and returns a structured comparison with citations and a recommendation.
Paste a stack trace or error message. The agent searches GitHub issues and StackOverflow for that exact error, scrapes the relevant threads, and returns a fix grounded in what developers who hit the same problem actually did.
Point at a docs URL. The agent scrapes the current API reference and writes working integration code from what is actually published — not from stale training data that may reference deprecated methods.
Give it your current version and target version. The agent scrapes the migration guide, extracts every breaking change with before/after patterns, and rewrites your code to match.
Turn any unstructured webpage — product listings, job posts, articles — into typed JSON matching a TypeScript interface, JSON schema, or database model you provide.
Auto-detects your stack (language, framework, AI toolkit) and writes a complete Olostep SDK integration: install commands, client setup, tool wiring, and a verification step.
Configure the Olostep API key and verify the connection. Includes troubleshooting for common setup issues.
## MCP Tools
The bundled MCP server (`olostep-mcp`) gives your agent 9 tools it can call directly. Use these when you need fine-grained control beyond what the workflow skills provide.
| Tool | What it does |
| --------------------- | ----------------------------------------------------------------------- |
| `scrape_website` | Scrape a single URL to markdown, HTML, JSON, or text |
| `get_webpage_content` | Fetch a webpage as clean, LLM-ready markdown |
| `search_web` | Search the live web, get AI-synthesized answers |
| `google_search` | Structured Google SERP data — organic results, PAA, knowledge graph |
| `answers` | Ask a question, get a cited answer with optional structured JSON output |
| `batch_scrape_urls` | Scrape up to 10,000 URLs in parallel with full rendering |
| `create_crawl` | Crawl a website by following links from a starting URL |
| `create_map` | Discover all URLs on a website, filterable by pattern and query |
| `get_website_urls` | Find and retrieve relevant URLs from a specific domain |
## What This Looks Like in Practice
### "Why is this failing?" — Debug from the live web
You paste `ECONNRESET when calling Stripe webhook endpoint` into your agent. It searches GitHub issues and StackOverflow for that exact error, scrapes the three most relevant threads, and returns a concrete fix — not "check your network settings," but the actual timeout configuration that solved it for other developers hitting the same wall.
### "Write the integration" — Code from current docs, not stale training data
You need to integrate a payment API. The `docs-to-code` skill scrapes the current API reference — the one published today, not the version your model was trained on six months ago — and writes working code using the parameters and endpoints that actually exist.
### "Which one should I use?" — Structured tool comparisons
Evaluating ORMs? Comparing auth providers? The `research` skill searches multiple sources, scrapes real pricing pages and feature matrices, and returns a structured comparison table with citations. You get a recommendation backed by what's actually on each product's website, not by training-data popularity.
### Build a RAG pipeline from any docs site
```
map → discover every URL on the docs site
batch → scrape all pages in parallel as clean markdown
→ feed into your vector store
```
Two skills, one pipeline. Works on documentation sites with hundreds or thousands of pages.
### Extract structured data at scale
```
map → find all product, listing, or job URLs
batch → scrape each page with a pre-built parser → typed JSON
→ pipe into your database, API, or seed files
```
### Migrate to a new framework version
You're upgrading Next.js 13 to 15. The `migrate-code` skill scrapes the official migration guide, extracts every breaking change with before/after code patterns, and rewrites your files to match — based on the real documentation, not on the model's best guess.
## Pre-built Parsers
Pass the `parser` parameter to any scrape call and get typed JSON back instead of raw content. No schema definition needed — these handle the extraction for you.
| Parser | Returns |
| ---------------------------- | ------------------------------------------------- |
| `@olostep/google-search` | Organic results, knowledge graph, People Also Ask |
| `@olostep/amazon-it-product` | Price, rating, features, availability |
| `@olostep/extract-emails` | Every email address found on the page |
| `@olostep/extract-calendars` | Structured calendar events |
| `@olostep/extract-socials` | Social media profile links |
## Links
Plugin listing, version history, and one-command install
500 free requests/month, no credit card
Full MCP server setup for Cursor, Claude Desktop, and other clients
Complete endpoint documentation with examples
# Olostep + ElizaOS Integration
Source: https://docs.olostep.com/integrations/eliza
Add Olostep web search to Eliza agents with the `OLOSTEP_SEARCH` action.
Eliza + Olostep gives your agents reliable web search, letting them look up current information, answer open-ended questions with live results, and return deduplicated links with titles and descriptions.
## Features
Adds the `OLOSTEP_SEARCH` action to Eliza agents for live web search.
Removes duplicate links and keeps the most relevant results at the top.
Configure one API key in your Eliza agent settings and start searching.
Works when users ask the agent to search the web, look something up, or find online sources.
Returns titles, descriptions, and URLs that are easy for agents to summarize or cite.
Calls Olostep directly through the `/searches` endpoint with standard `fetch`.
## Installation
```bash npm theme={null}
npm install @olostep/plugin-elizaos-olostep
```
```bash pnpm theme={null}
pnpm add @olostep/plugin-elizaos-olostep
```
```bash bun theme={null}
bun add @olostep/plugin-elizaos-olostep
```
This package is published on npm as `@olostep/plugin-elizaos-olostep`.
## Setup
1. Create an Olostep API key in your Olostep dashboard.
2. Add the key to your Eliza agent settings as `OLOSTEP_API_KEY`.
3. Include the plugin in your character config.
```json theme={null}
{
"name": "MyAgent",
"settings": {
"secrets": {
"OLOSTEP_API_KEY": "your-olostep-api-key-here"
}
}
}
```
```typescript TypeScript theme={null}
import type { Character } from '@elizaos/core';
export const character: Character = {
name: 'MyAgent',
plugins: [
'@elizaos/plugin-bootstrap',
'@elizaos/plugin-openai',
'@olostep/plugin-elizaos-olostep',
],
};
```
```json JSON theme={null}
{
"name": "MyAgent",
"plugins": [
"@elizaos/plugin-bootstrap",
"@elizaos/plugin-openai",
"@olostep/plugin-elizaos-olostep"
]
}
```
## Available Tools
### `OLOSTEP_SEARCH`
Searches the web with Olostep and returns a list of relevant links with titles and descriptions. Use it when the user asks the agent to search for information, look up a topic, or find current web sources.
The Olostep API key stored in the agent runtime secrets.
The search query. Eliza uses the incoming user message text as the query.
```typescript Basic Setup theme={null}
// Register the plugin and let Eliza route search requests
import type { Character } from '@elizaos/core';
export const character: Character = {
name: 'ResearchAgent',
plugins: ['@olostep/plugin-elizaos-olostep'],
};
```
```typescript Advanced Setup theme={null}
// Combine Olostep with a model plugin for a full research agent
import type { Character } from '@elizaos/core';
export const character: Character = {
name: 'ResearchAgent',
bio: ['Investigates current events and summarizes web sources.'],
plugins: [
'@elizaos/plugin-bootstrap',
'@elizaos/plugin-openai',
'@olostep/plugin-elizaos-olostep',
],
};
```
```typescript With Style Guide theme={null}
// Tailor the agent to prefer web search
import type { Character } from '@elizaos/core';
export const character: Character = {
name: 'NewsAgent',
style: {
all: ['Use web search when the answer may have changed recently.'],
},
plugins: ['@olostep/plugin-elizaos-olostep'],
};
```
The action returns structured search results in `data.links`, and the agent response includes a readable summary with up to five top links.
## Full Agent Examples
### Research Assistant
A general-purpose research agent that fetches recent facts before answering:
```typescript theme={null}
import type { Character } from '@elizaos/core';
export const character: Character = {
name: 'ResearchAssistant',
bio: [
'Answers questions using current web sources.',
'Summarizes links into concise, cited responses.',
],
plugins: [
'@elizaos/plugin-bootstrap',
'@elizaos/plugin-openai',
'@olostep/plugin-elizaos-olostep',
],
settings: {
secrets: {
OLOSTEP_API_KEY: process.env.OLOSTEP_API_KEY!,
},
},
};
```
### News Monitor
An agent that tracks timely topics and reports notable updates:
```typescript theme={null}
import type { Character } from '@elizaos/core';
export const character: Character = {
name: 'NewsMonitor',
bio: ['Tracks timely topics and reports notable updates from the web.'],
plugins: [
'@elizaos/plugin-bootstrap',
'@elizaos/plugin-openai',
'@olostep/plugin-elizaos-olostep',
],
style: {
all: ['Prefer current sources and include direct URLs when possible.'],
},
};
```
Use this for alerts, market watch tasks, trend research, and other time-sensitive workflows.
### Support Agent with Search Fallback
Perfect for answering customer questions with product documentation lookup:
```typescript theme={null}
import type { Character } from '@elizaos/core';
export const character: Character = {
name: 'SupportAgent',
plugins: [
'@elizaos/plugin-bootstrap',
'@elizaos/plugin-openai',
'@olostep/plugin-elizaos-olostep',
],
topics: [
'product support',
'documentation lookup',
'release notes search',
],
};
```
This pattern works well when your agent should search docs or product pages before answering a customer question.
## Configuration
### Enable the plugin
Add `@olostep/plugin-elizaos-olostep` to the `plugins` array in your character config.
### Disable web search
Remove the plugin from the character config if you want an Eliza agent that does not have Olostep search access.
### Use only some capabilities
This plugin exposes a single action, so there is no per-tool toggle. Control behavior through:
* Which plugins you load in the character config
* The agent instructions and style
* When your runtime injects `OLOSTEP_API_KEY`
## Specialized Features
* **Direct `/searches` endpoint access** — the plugin calls Olostep directly with `fetch`.
* **Result deduplication** — duplicate URLs are removed before the response is returned.
* **Friendly fallbacks** — the action returns clear errors when the API key is missing or the query is empty.
* **Top-result limiting** — responses are trimmed to the five most relevant links.
## Pricing
Pricing for search usage depends on your Olostep plan and dashboard settings.
* Check your Olostep dashboard for current usage and billing details.
* Review your account limits before deploying high-volume agents.
## Support
* **NPM Package**: [@olostep/plugin-elizaos-olostep](https://www.npmjs.com/package/@olostep/plugin-elizaos-olostep)
* **Olostep Website**: [olostep.com](https://www.olostep.com)
* **Olostep Dashboard**: [dashboard.olostep.com](https://www.olostep.com/dashboard)
* **ElizaOS**: [elizaos.ai](https://elizaos.ai)
* **Email Support**: [info@olostep.com](mailto:info@olostep.com)
## Related Resources
Learn how the search endpoint returns web results
Queue searches and other jobs for larger workflows
Generate answer-style outputs from retrieved web sources
Explore deeper site collection and crawling workflows
Use the Python SDK for custom automation around Olostep
Build JavaScript integrations and agent workflows
# Olostep + Kilo Integration
Source: https://docs.olostep.com/integrations/kilo
Use Olostep MCP Server as an external tool in Kilo for web search, scraping, and content extraction
Kilo is the all-in-one agentic engineering platform with support for MCP (Model Context Protocol) servers. The Olostep MCP Server integrates seamlessly with Kilo, giving your AI agent powerful web data capabilities.
## Features
The Olostep MCP Server provides access to 5 core Olostep capabilities:
Extract content from any single URL in multiple formats (Markdown, HTML, JSON, text)
Search the web and get structured, parser-based results
Get AI-powered answers with natural language queries and citations
Process up to 10,000 URLs in parallel. Perfect for large-scale data extraction
Autonomously discover and scrape entire websites by following links
## Installation
### 1. Get Your API Key
Get your Olostep API key from the [Olostep Dashboard](https://olostep.com/dashboard/api-keys).
### 2. Configure Kilo
In your project directory, create or edit `.kilocode/mcp.json` to add the Olostep MCP Server:
```json Remote Hosted MCP (Recommended) theme={null}
{
"mcpServers": {
"olostep": {
"url": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_OLOSTEP_API_KEY"
}
}
}
}
```
```json Local/Stdio MCP theme={null}
{
"mcpServers": {
"olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_OLOSTEP_API_KEY"
}
}
}
}
```
Replace `YOUR_OLOSTEP_API_KEY` with your actual API key from the Olostep Dashboard.
### 3. Verify Installation
Run Kilo and the Olostep tools should be automatically available:
```bash theme={null}
kilo
```
## Available Tools
Once configured, the following Olostep tools are available to your Kilo agent:
### scrape\_website
Extract content from a single URL. Supports multiple formats and JavaScript rendering.
**Example prompt:**
```
Scrape the content from https://example.com and give me the markdown version
```
**Parameters:**
* `url` - Website URL to scrape (required)
* `format` - Output format: `markdown`, `html`, `json`, or `text` (default: markdown)
* `country` - Country code for location-specific content (e.g., "US", "GB")
* `wait_before_scraping` - Wait time in milliseconds for JavaScript rendering (0-10000)
* `parser` - Optional parser ID for specialized extraction (e.g., "@olostep/amazon-product")
### search\_web
Search the web and return structured, parser-based results.
**Example prompt:**
```
Search for the latest news about AI regulations in the EU
```
**Parameters:**
* `query` - Search query (required)
* `country` - Country code for location-specific results
### answers
Get AI-powered answers from web searches with sources and citations.
**Example prompt:**
```
What are the top 5 emerging AI trends in 2026? Get me structured answers with sources.
```
**Parameters:**
* `query` - Natural language question (required)
* `format` - Output format for answers (JSON structure of your choice)
* `country` - Country code for location-specific searches
### batch\_scrape\_urls
Process multiple URLs in parallel. Perfect for large-scale data extraction.
**Example prompt:**
```
Scrape these 50 product URLs and extract title, price, and description in JSON format
```
**Parameters:**
* `urls` - Array of URLs to scrape (required)
* `format` - Output format for all URLs
* `country` - Country code for location-specific content
* `parser` - Optional parser ID for all URLs
### create\_crawl
Autonomously discover and scrape entire websites by following links from a start URL.
**Example prompt:**
```
Crawl the website starting from https://example.com and extract all product pages
```
**Parameters:**
* `start_url` - Starting URL for the crawl (required)
* `max_pages` - Maximum number of pages to crawl (default: 10). Set to `1` to scrape only the start URL.
### create\_map
Extract all URLs from a website for site structure analysis and content discovery.
**Example prompt:**
```
Map all URLs on https://example.com to understand the site structure
```
**Parameters:**
* `website_url` - Website to map (required)
* `search_query` - Optional query to filter results
* `top_n` - Limit number of returned URLs
## Example Workflows
### Research Task with Web Search
Let Kilo agent gather information from the web:
```
Use the search_web tool to find recent articles about LLM safety,
then use the answers tool to synthesize the key findings with sources.
Give me a comprehensive summary.
```
### Product Data Extraction
Scrape multiple product pages:
```
I have 100 product URLs from an e-commerce site. Use batch_scrape_urls
to extract title, price, description, and availability from each one.
Format the results as JSON.
```
### Website Discovery and Analysis
Crawl and analyze a website structure:
```
I want to understand the structure of https://example.com.
First, use create_map to get all URLs, then crawl the main sections
and give me a summary of the site's content organization.
```
## Environment Variables
When using the local/stdio configuration, make sure your environment has the API key set:
```bash theme={null}
export OLOSTEP_API_KEY="your_api_key_here"
kilo
```
Or in your `.env` file:
```
OLOSTEP_API_KEY=your_api_key_here
```
## Troubleshooting
**Tools not appearing in Kilo:**
* Verify the `.kilocode/mcp.json` is properly formatted (valid JSON)
* Ensure your API key is correct in the configuration
* Try restarting Kilo after configuration changes
**API key authentication errors:**
* Double-check your API key from the [Olostep Dashboard](https://olostep.com/dashboard/api-keys)
* Make sure there are no extra spaces or special characters in the key
**Remote hosted MCP not connecting:**
* Verify your internet connection
* Check that `https://mcp.olostep.com/mcp` is accessible
* Ensure the Authorization header format is correct: `Bearer YOUR_KEY`
## Learn More
* [Kilo Website](https://kilo.ai/)
* [Kilo Documentation](https://docs.kilo.ai/)
* [Olostep MCP Server Repository](https://github.com/olostep/olostep-mcp-server)
* [Model Context Protocol (MCP) Specification](https://modelcontextprotocol.io/)
# Olostep + LangChain Integration
Source: https://docs.olostep.com/integrations/langchain
Build intelligent AI agents with web scraping and search capabilities using LangChain
The Olostep LangChain integration provides comprehensive tools to build AI agents that can search, scrape, analyze, and structure data from any website. Perfect for LangChain and LangGraph applications.
## Features
The integration provides access to all 5 Olostep API capabilities:
Extract content from any single URL in multiple formats (Markdown, HTML, JSON, text)
Process up to 10,000 URLs in parallel. Batch jobs complete in 5-8 minutes
AI-powered web search with natural language queries and structured output
Extract all URLs from a website for site structure analysis
Autonomously discover and scrape entire websites by following links
## Installation
```bash pip theme={null}
pip install langchain-olostep
```
```bash poetry theme={null}
poetry add langchain-olostep
```
## Setup
Set your Olostep API key as an environment variable:
```bash theme={null}
export OLOSTEP_API_KEY="your_olostep_api_key_here"
```
Get your API key from the [Olostep Dashboard](https://olostep.com/dashboard).
## Available Tools
### scrape\_website
Extract content from a single URL. Supports multiple formats and JavaScript rendering.
Website URL to scrape (must include http\:// or https\://)
Output format: `markdown`, `html`, `json`, or `text`
Country code for location-specific content (e.g., "US", "GB", "CA")
Wait time in milliseconds for JavaScript rendering (0-10000)
Optional parser ID for specialized extraction (e.g., "@olostep/amazon-product")
```python Basic Scraping theme={null}
from langchain_olostep import scrape_website
import asyncio
# Scrape a website
content = asyncio.run(scrape_website.ainvoke({
"url": "https://example.com",
"format": "markdown"
}))
print(content)
```
```python With JavaScript theme={null}
# Wait for dynamic content
content = asyncio.run(scrape_website.ainvoke({
"url": "https://example.com",
"format": "markdown",
"wait_before_scraping": 2000
}))
```
```python With Parser theme={null}
# Use specialized parser
content = asyncio.run(scrape_website.ainvoke({
"url": "https://www.amazon.com/dp/PRODUCT_ID",
"parser": "@olostep/amazon-product",
"format": "json"
}))
```
### scrape\_batch
Process multiple URLs in parallel (up to 10,000 at once).
List of URLs to scrape
Output format for all URLs: `markdown`, `html`, `json`, or `text`
Country code for location-specific content
Wait time in milliseconds for JavaScript rendering
Optional parser ID for specialized extraction
```python Batch Scraping theme={null}
from langchain_olostep import scrape_batch
import asyncio
# Scrape multiple URLs
result = asyncio.run(scrape_batch.ainvoke({
"urls": [
"https://example1.com",
"https://example2.com",
"https://example3.com"
],
"format": "markdown"
}))
print(result)
# Returns: {"batch_id": "batch_xxx", "status": "in_progress", ...}
```
### answer\_question
Search the web and get AI-powered answers with sources. Perfect for data enrichment and research.
Question or task to search for
Optional JSON schema dict/string describing desired output format
```python Simple Question theme={null}
from langchain_olostep import answer_question
import asyncio
# Ask a simple question
result = asyncio.run(answer_question.ainvoke({
"task": "What is the capital of France?"
}))
print(result)
# Returns: {"answer": {"result": "Paris"}, "sources": [...]}
```
```python Structured Output theme={null}
# Get structured data with JSON schema
result = asyncio.run(answer_question.ainvoke({
"task": "What is the latest book by J.K. Rowling?",
"json_schema": {
"book_title": "",
"author": "",
"release_date": ""
}
}))
print(result)
# Returns structured answer with sources
```
```python Data Enrichment theme={null}
# Enrich company data
result = asyncio.run(answer_question.ainvoke({
"task": "Find the CEO and headquarters of Stripe",
"json_schema": {
"ceo_name": "",
"headquarters": "",
"founded_year": ""
}
}))
# Handles uncertainty with "NOT_FOUND" values
```
### extract\_urls
Extract all URLs from a website for site structure analysis.
Website URL to extract URLs from
Optional search query to filter URLs
Limit the number of URLs returned
Glob patterns to include (e.g., \["/blog/\*\*"])
Glob patterns to exclude (e.g., \["/admin/\*\*"])
```python Extract All URLs theme={null}
from langchain_olostep import extract_urls
import asyncio
# Get all URLs from a website
result = asyncio.run(extract_urls.ainvoke({
"url": "https://example.com",
"top_n": 100
}))
print(result)
# Returns: {"urls": [...], "total_urls": 100, ...}
```
```python Filter URLs theme={null}
# Get only blog URLs
result = asyncio.run(extract_urls.ainvoke({
"url": "https://example.com",
"include_urls": ["/blog/**"],
"exclude_urls": ["/admin/**", "/private/**"],
"top_n": 50
}))
```
### crawl\_website
Autonomously discover and scrape entire websites by following links.
Starting URL for the crawl
Maximum number of pages to crawl
Glob patterns to include (e.g., \["/\*\*"] for all)
Glob patterns to exclude (e.g., \["/admin/\*\*"])
Maximum depth to crawl from start\_url
Include external URLs
```python Crawl Website theme={null}
from langchain_olostep import crawl_website
import asyncio
# Crawl entire documentation site
result = asyncio.run(crawl_website.ainvoke({
"start_url": "https://docs.example.com",
"max_pages": 100
}))
print(result)
# Returns: {"crawl_id": "crawl_xxx", "status": "in_progress", ...}
```
```python With Filters theme={null}
# Crawl with URL filters
result = asyncio.run(crawl_website.ainvoke({
"start_url": "https://example.com",
"max_pages": 200,
"include_urls": ["/**"],
"exclude_urls": ["/admin/**", "/private/**"],
"max_depth": 3
}))
```
## LangChain Agent Integration
Build intelligent agents that can search and scrape the web:
```python Basic Agent theme={null}
from langchain.agents import initialize_agent, AgentType
from langchain_openai import ChatOpenAI
from langchain_olostep import (
scrape_website,
answer_question,
extract_urls
)
# Create agent with Olostep tools
tools = [scrape_website, answer_question, extract_urls]
llm = ChatOpenAI(model="gpt-4o-mini")
agent = initialize_agent(
tools=tools,
llm=llm,
agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
verbose=True
)
# Use the agent
result = agent.run("""
Research the company at https://company.com:
1. Scrape their about page
2. Search for their latest funding round
3. Extract all their product pages
""")
print(result)
```
## LangGraph Integration
Build complex multi-step workflows with LangGraph:
```python Research Agent theme={null}
from langgraph.graph import StateGraph, END
from langchain_olostep import (
scrape_website,
scrape_batch,
answer_question,
extract_urls
)
from langchain_openai import ChatOpenAI
import json
def create_research_agent():
workflow = StateGraph(dict)
def discover_pages(state):
# Extract all URLs from target site
result = extract_urls.invoke({
"url": state["target_url"],
"include_urls": ["/product/**"],
"top_n": 50
})
state["urls"] = json.loads(result)["urls"]
return state
def scrape_pages(state):
# Scrape discovered pages in batch
result = scrape_batch.invoke({
"urls": state["urls"],
"format": "markdown"
})
state["batch_id"] = json.loads(result)["batch_id"]
return state
def answer_questions(state):
# Use AI to answer questions about the data
result = answer_question.invoke({
"task": state["research_question"],
"json_schema": state["desired_format"]
})
state["answer"] = json.loads(result)["answer"]
return state
workflow.add_node("discover", discover_pages)
workflow.add_node("scrape", scrape_pages)
workflow.add_node("analyze", answer_questions)
workflow.set_entry_point("discover")
workflow.add_edge("discover", "scrape")
workflow.add_edge("scrape", "analyze")
workflow.add_edge("analyze", END)
return workflow.compile()
# Use the agent
agent = create_research_agent()
result = agent.invoke({
"target_url": "https://store.com",
"research_question": "What are the top 5 most expensive products?",
"desired_format": {
"products": [{"name": "", "price": "", "url": ""}]
}
})
```
## Advanced Use Cases
### Data Enrichment
Enrich spreadsheet data with web information:
```python theme={null}
from langchain_olostep import answer_question
companies = ["Stripe", "Shopify", "Square"]
for company in companies:
result = answer_question.invoke({
"task": f"Find information about {company}",
"json_schema": {
"ceo": "",
"headquarters": "",
"employee_count": "",
"latest_funding": ""
}
})
print(f"{company}: {result}")
```
### E-commerce Product Scraping
Scrape product data with specialized parsers:
```python theme={null}
from langchain_olostep import scrape_website
# Scrape Amazon product
result = scrape_website.invoke({
"url": "https://www.amazon.com/dp/PRODUCT_ID",
"parser": "@olostep/amazon-product",
"format": "json"
})
# Returns structured product data: price, title, rating, etc.
```
### SEO Audit
Analyze entire websites for SEO:
```python theme={null}
from langchain_olostep import extract_urls, scrape_batch
import json
# 1. Discover all pages
urls_result = extract_urls.invoke({
"url": "https://yoursite.com",
"top_n": 1000
})
# 2. Scrape all pages
urls = json.loads(urls_result)["urls"]
batch_result = scrape_batch.invoke({
"urls": urls,
"format": "html"
})
```
### Documentation Scraping
Crawl and extract documentation:
```python theme={null}
from langchain_olostep import crawl_website
# Crawl entire docs site
result = crawl_website.invoke({
"start_url": "https://docs.example.com",
"max_pages": 500,
"include_urls": ["/docs/**"],
"exclude_urls": ["/api/**", "/v1/**"]
})
```
## Specialized Parsers
Olostep provides pre-built parsers for popular websites:
* `@olostep/google-search` - Google search results
Use them with the `parser` parameter:
```python theme={null}
scrape_website.invoke({
"url": "https://www.google.com/search?q=alexander+the+great&gl=us&hl=en",
"parser": "@olostep/google-search"
})
```
## Error Handling
```python theme={null}
from langchain_core.exceptions import LangChainException
try:
result = await scrape_website.ainvoke({
"url": "https://example.com"
})
except LangChainException as e:
print(f"Scraping failed: {e}")
```
## Best Practices
When scraping more than 3-5 URLs, use `scrape_batch` instead of multiple `scrape_website` calls. Batch processing is much faster and more cost-effective.
For JavaScript-heavy sites, use `wait_before_scraping` parameter (2000-5000ms is typical). This ensures dynamic content is fully loaded.
For popular websites (Amazon, LinkedIn, Google), use our pre-built parsers to get structured data automatically.
When using `extract_urls` or `crawl_website`, use glob patterns to focus on relevant pages and avoid unnecessary processing.
Implement exponential backoff for rate limit errors. The API automatically handles most rate limiting internally.
## Support
* **PyPI Package**: [langchain-olostep](https://pypi.org/project/langchain-olostep/)
* **Documentation**: [docs.olostep.com](https://docs.olostep.com)
* **Issues**: [GitHub Issues](https://github.com/olostep/langchain-olostep/issues)
* **Email**: [info@olostep.com](mailto:info@olostep.com)
## Related Resources
Learn about the Scrapes endpoint
Learn about the Batches endpoint
Learn about the Answers endpoint
Learn about the Maps endpoint
Learn about the Crawls endpoint
Explore the Python SDK
LangChain platform
# Olostep + Mastra Integration
Source: https://docs.olostep.com/integrations/mastra
Build AI agents with web search, scraping and crawling capabilities using Mastra.ai's agent framework
The Olostep Mastra integration brings powerful web data extraction capabilities to Mastra.ai agents. Olostep is a Web search, scraping and crawling API — an API to search, extract and structure web data. Build intelligent AI agents that can autonomously search, scrape, analyze, and structure data from any website.
[Install from npm →](https://www.npmjs.com/package/@olostep/mastra-tools)
## Features
The integration provides 4 powerful APIs for automated web data extraction:
Extract content from any single URL in multiple formats (Markdown, HTML, JSON, text)
Process up to 100,000 URLs in parallel. Perfect for large-scale data extraction
Autonomously discover and scrape entire websites by following links
Extract all URLs from a website for site structure analysis and content discovery
## Installation
```bash npm theme={null}
npm install @olostep/mastra-tools
```
```bash yarn theme={null}
yarn add @olostep/mastra-tools
```
```bash pnpm theme={null}
pnpm add @olostep/mastra-tools
```
## Setup
### 1. Install the Package
```bash theme={null}
npm install @olostep/mastra-tools @mastra/core
```
### 2. Import and Register Integration
In your Mastra configuration file:
```typescript theme={null}
import { Mastra } from '@mastra/core';
import { createOlostepIntegration } from '@olostep/mastra-tools';
// Create the Olostep integration
const olostep = createOlostepIntegration();
// Register APIs (this makes them available to agents)
olostep.registerApis();
// Add to your Mastra config
export const mastra = new Mastra({
config: {
integrations: [olostep],
// ... other config
},
});
```
### 3. Configure API Key
Set your Olostep API key as an environment variable:
```bash theme={null}
export OLOSTEP_API_KEY=your-api-key-here
```
Or in your `.env` file:
```
OLOSTEP_API_KEY=your-api-key-here
```
Get your API key from the [Olostep Dashboard](https://olostep.com/dashboard).
## Available APIs
The integration exposes 4 APIs that your Mastra agents can use:
### scrapeWebsite
Extract content from a single URL. Supports multiple formats and JavaScript rendering.
**Use Cases:**
* Monitor specific pages for changes
* Extract product information from e-commerce sites
* Gather data from news articles or blog posts
* Pull content for content aggregation
**Schema Parameters:**
Your Olostep API key
Website URL to scrape (must include http\:// or https\://)
Output formats: \['html', 'markdown', 'json', 'text']
Country code for location-specific content (e.g., "US", "GB", "CA")
Wait time in milliseconds for JavaScript rendering (0-10000)
Optional parser ID for specialized extraction (e.g., "@olostep/amazon-product")
**Response:**
* `id` - Scrape ID
* `url_to_scrape` - Scraped URL
* `result.markdown_content` - Markdown content
* `result.html_content` - HTML content
* `result.json_content` - JSON content
* `result.text_content` - Text content
* `result.screenshot_hosted_url` - Screenshot URL (if available)
* `result.markdown_hosted_url` - Hosted markdown URL
* `object` - Object type ("scrape")
* `created` - Unix timestamp
**Example Usage:**
```typescript theme={null}
// In your agent or workflow
const result = await mastra.callApi({
integrationName: 'olostep',
api: 'scrapeWebsite',
payload: {
data: {
apiKey: process.env.OLOSTEP_API_KEY,
url_to_scrape: 'https://example.com',
formats: ['markdown'],
country: 'US',
}
}
});
```
### batchScrape
Process multiple URLs in parallel (up to 100,000 at once). Perfect for large-scale data extraction.
**Use Cases:**
* Scrape entire product catalogs
* Extract data from multiple search results
* Process lists of URLs from spreadsheets
* Bulk content extraction
**Schema Parameters:**
Your Olostep API key
Array of objects with `url` and optional `custom_id` fields
Example: `[{"url":"https://example.com","custom_id":"site1"}]`
Output formats for all URLs
Country code for location-specific scraping
Wait time in milliseconds for JavaScript rendering
Optional parser ID for specialized extraction
**Response:**
* `batch_id` - Batch ID (use this to retrieve results later)
* `status` - Processing status
* `object` - Object type ("batch")
**Example Usage:**
```typescript theme={null}
const result = await mastra.callApi({
integrationName: 'olostep',
api: 'batchScrape',
payload: {
data: {
apiKey: process.env.OLOSTEP_API_KEY,
batch_array: [
{ url: 'https://example.com', custom_id: 'site1' },
{ url: 'https://test.com', custom_id: 'site2' },
],
formats: ['markdown'],
}
}
});
```
### createCrawl
Autonomously discover and scrape entire websites by following links. Perfect for documentation sites, blogs, and content repositories.
**Use Cases:**
* Crawl and archive entire documentation sites
* Extract all blog posts from a website
* Build knowledge bases from web content
* Monitor website structure changes
**Schema Parameters:**
Your Olostep API key
Starting URL for the crawl (must include http\:// or https\://)
Maximum number of pages to crawl. Set to `1` to scrape only the start URL.
Format for scraped content
Optional country code for location-specific crawling
Optional parser ID for specialized content extraction
**Response:**
* `id` - Crawl ID (use this to retrieve results later)
* `object` - Object type ("crawl")
* `status` - Crawl status
* `created` - Unix timestamp
**Example Usage:**
```typescript theme={null}
const result = await mastra.callApi({
integrationName: 'olostep',
api: 'createCrawl',
payload: {
data: {
apiKey: process.env.OLOSTEP_API_KEY,
start_url: 'https://docs.example.com',
max_pages: 50,
formats: ['markdown'],
}
}
});
```
### createMap
Extract all URLs from a website for content discovery and site structure analysis.
**Use Cases:**
* Build sitemaps and site structure diagrams
* Discover all pages before batch scraping
* Find broken or missing pages
* SEO audits and analysis
**Schema Parameters:**
Your Olostep API key
Website URL to extract links from (must include http\:// or https\://)
Optional search query to filter URLs (e.g., "blog")
Limit the number of URLs returned
Glob patterns to include specific paths (e.g., \["/blog/\*\*"])
Glob patterns to exclude specific paths (e.g., \["/admin/\*\*"])
**Response:**
* `id` - Map ID
* `object` - Object type ("map")
* `url` - Website URL
* `total_urls` - Total URLs found
* `urls` - Array of discovered URLs
**Example Usage:**
```typescript theme={null}
const result = await mastra.callApi({
integrationName: 'olostep',
api: 'createMap',
payload: {
data: {
apiKey: process.env.OLOSTEP_API_KEY,
url: 'https://example.com',
search_query: 'blog',
top_n: 100,
include_urls: ['/blog/**'],
}
}
});
```
## Using with Agents
### Basic Agent Example
Create an agent that can scrape websites:
```typescript theme={null}
import { Agent } from '@mastra/core';
import { createOlostepIntegration } from '@olostep/mastra-tools';
const olostep = createOlostepIntegration();
olostep.registerApis();
const agent = new Agent({
name: 'web-researcher',
instructions: `
You are a web research assistant. When users ask you to get information from a website,
use the Olostep scrapeWebsite API to extract the content, then summarize it for them.
`,
model: 'openai/gpt-4',
});
// The agent can now use Olostep APIs through Mastra's API system
```
### Agent Workflow Example
Build a research workflow that discovers and scrapes content:
```typescript theme={null}
// 1. Map a website to discover URLs
const mapResult = await mastra.callApi({
integrationName: 'olostep',
api: 'createMap',
payload: {
data: {
apiKey: process.env.OLOSTEP_API_KEY,
url: 'https://example.com',
include_urls: ['/blog/**'],
}
}
});
// 2. Batch scrape discovered URLs
const batchResult = await mastra.callApi({
integrationName: 'olostep',
api: 'batchScrape',
payload: {
data: {
apiKey: process.env.OLOSTEP_API_KEY,
batch_array: mapResult.urls.slice(0, 10).map(url => ({ url })),
formats: ['markdown'],
}
}
});
// 3. Process results with your agent
const summary = await agent.generate({
messages: [{
role: 'user',
content: `Summarize this content: ${batchResult.result.markdown_content}`
}]
});
```
## Popular Use Cases
### Research Agent
Build an agent that autonomously researches topics:
**Workflow:**
1. User asks: "Research AI trends"
2. Agent uses `createMap` to discover relevant pages
3. Agent uses `batchScrape` to extract content
4. Agent analyzes and summarizes findings
5. Returns structured research report
**Workflow:**
1. Schedule daily monitoring
2. Use `scrapeWebsite` to check competitor pages
3. Compare with previous data
4. Alert on significant changes
5. Generate weekly reports
**Workflow:**
1. Use `createCrawl` to discover all blog posts
2. Use `batchScrape` to extract content
3. Process with AI to extract key topics
4. Store in knowledge base
5. Generate content calendar
### E-commerce Intelligence
Monitor products and prices:
```
Agent Workflow:
1. Scrape product pages (scrapeWebsite)
2. Extract structured data (with parser)
3. Track price changes
4. Generate alerts
5. Update database
```
### SEO Analysis
Analyze website structure and content:
```
Agent Workflow:
1. Map website structure (createMap)
2. Crawl important sections (createCrawl)
3. Analyze content quality
4. Identify SEO opportunities
5. Generate recommendations
```
## Specialized Parsers
Olostep provides pre-built parsers for popular websites. Use them with the `parser` parameter:
`@olostep/google-search`
Extract: search results, titles, snippets, URLs
`@olostep/google-maps`
Extract: business info, reviews, ratings, location
### Using Parsers
Add the parser ID to the `parser` parameter:
```typescript theme={null}
const result = await mastra.callApi({
integrationName: 'olostep',
api: 'scrapeWebsite',
payload: {
data: {
apiKey: process.env.OLOSTEP_API_KEY,
url_to_scrape: 'https://www.amazon.com/dp/PRODUCT_ID',
formats: ['json'],
parser: '@olostep/amazon-product',
}
}
});
```
The parser automatically extracts structured data specific to that website type.
## Best Practices
When scraping more than 3-5 URLs, use `batchScrape` instead of multiple `scrapeWebsite` calls. Batch processing is:
* Much faster (parallel processing)
* More cost-effective
* Easier to manage
* Better for rate limits
For JavaScript-heavy sites, use the `wait_before_scraping` parameter:
* Simple sites: 0-1000ms
* Dynamic sites: 2000-3000ms
* Heavy JavaScript: 5000-8000ms
Test with different values to find the optimal wait time.
For popular websites (Amazon, LinkedIn, Google), use pre-built parsers:
* Get structured data automatically
* More reliable extraction
* No need for custom parsing
* Maintained by Olostep
Batch, Crawl, and Map operations are asynchronous:
* Store the returned ID (batch\_id, crawl\_id, map\_id)
* Poll for completion or use webhooks
* Set up separate workflows for retrieval
Always wrap API calls in try-catch blocks:
```typescript theme={null}
try {
const result = await mastra.callApi({
integrationName: 'olostep',
api: 'scrapeWebsite',
payload: { data: {...} }
});
} catch (error) {
// Handle authentication, rate limit, or network errors
console.error('Scraping failed:', error.message);
}
```
Be mindful of rate limits:
* Space out requests with delays
* Use batch processing when possible
* Monitor usage in Olostep dashboard
* Upgrade plan if needed
## Complete Example
Here's a complete example of building a research agent:
```typescript theme={null}
import { Mastra } from '@mastra/core';
import { Agent } from '@mastra/core';
import { createOlostepIntegration } from '@olostep/mastra-tools';
// Create and register Olostep integration
const olostep = createOlostepIntegration();
olostep.registerApis();
// Initialize Mastra
export const mastra = new Mastra({
config: {
integrations: [olostep],
// ... other config
},
});
// Create research agent
const researchAgent = new Agent({
name: 'research-assistant',
instructions: `
You are a research assistant that can search, extract, and structure web data.
When users ask you to research a topic:
1. Use Olostep's createMap to discover relevant pages
2. Use batchScrape to extract content from multiple sources
3. Analyze and summarize the findings
4. Present structured research reports
`,
model: 'openai/gpt-4',
});
// Use the agent
async function researchTopic(topic: string) {
// Step 1: Discover relevant pages
const mapResult = await mastra.callApi({
integrationName: 'olostep',
api: 'createMap',
payload: {
data: {
apiKey: process.env.OLOSTEP_API_KEY!,
url: `https://example.com/search?q=${topic}`,
top_n: 20,
}
}
});
// Step 2: Scrape discovered pages
const batchResult = await mastra.callApi({
integrationName: 'olostep',
api: 'batchScrape',
payload: {
data: {
apiKey: process.env.OLOSTEP_API_KEY!,
batch_array: mapResult.urls.slice(0, 10).map(url => ({ url })),
formats: ['markdown'],
}
}
});
// Step 3: Analyze with agent
const summary = await researchAgent.generate({
messages: [{
role: 'user',
content: `Based on this research data, provide a comprehensive summary of ${topic}`
}]
});
return summary;
}
```
## Troubleshooting
**Error**: "Invalid API key"
**Solutions**:
* Check API key from [dashboard](https://olostep.com/dashboard)
* Ensure API key is set in environment variable
* Verify API key is active
* Check for extra spaces in API key
**Error**: "API not found" or "Integration not registered"
**Solutions**:
* Ensure `registerApis()` is called after creating integration
* Verify integration is added to Mastra config
* Check integration name is 'olostep'
* Restart Mastra server after changes
**Error**: Content fields are empty
**Solutions**:
* Increase `wait_before_scraping` time
* Check if website requires login
* Try different format (HTML vs Markdown)
* Verify URL is accessible
* Check if site blocks automated access
**Error**: "Rate limit exceeded"
**Solutions**:
* Space out requests with delays
* Use batch processing instead of individual scrapes
* Upgrade your Olostep plan
* Check rate limit in dashboard
**Error**: Module not found or type errors
**Solutions**:
* Ensure `@mastra/core` is installed
* Check TypeScript version compatibility
* Verify all dependencies are installed
* Rebuild: `npm run build`
## Pricing
Olostep charges based on API usage, independent of Mastra:
* **Scrapes**: Pay per scrape
* **Batches**: Pay per URL in batch
* **Crawls**: Pay per page crawled
* **Maps**: Pay per map operation
Check current pricing at [olostep.com/pricing](https://www.olostep.com/pricing).
## Support
Need help with the Mastra integration?
Browse complete API docs
Email: [info@olostep.com](mailto:info@olostep.com)
Learn about Mastra framework
## Related Resources
Learn about the Scrapes endpoint
Learn about the Batches endpoint
Learn about the Crawls endpoint
Learn about the Maps endpoint
Automate with Zapier workflows
Build AI agents with LangChain
Mastra platform
## Get Started
Ready to build AI agents with web scraping capabilities?
Install @olostep/mastra-tools from npm
Build intelligent AI agents that can search, extract, and structure web data with Olostep and Mastra!
# Olostep MCP Server
Source: https://docs.olostep.com/integrations/mcp-server
Give any MCP-compatible AI client web scraping, search, crawling, and AI-answer tools in under a minute
The Olostep MCP server gives any MCP-compatible AI client (Claude, Cursor, Windsurf, VS Code, Claude Code, etc.) 10 ready-to-use tools for the live web — scraping, search, AI answers with citations, batch jobs, site crawling, and URL discovery.
Pull markdown, HTML, JSON, or text from any URL with optional JS rendering
Web-grounded answers with sources and structured output
Up to 10k URLs in parallel, or autonomously discover a whole site
Find every URL on a site, or run parser-based web search
## Before you start
You need an Olostep API key. Get one from the [Olostep dashboard](https://www.olostep.com/dashboard/api-keys) — the free tier covers personal use.
## Pick a setup path
The fastest path for every client is the **hosted endpoint** at `https://mcp.olostep.com/mcp`. No installs, no Node, no Docker — just paste a URL and your API key.
If you need it to run fully local (offline use, corporate proxy, air-gapped), every client also supports a **local stdio** install via `npx`. Each section below shows both.
**Hosted endpoint** uses `Authorization: Bearer YOUR_API_KEY`. **Local stdio** uses `OLOSTEP_API_KEY` as an environment variable. Don't mix them up — wrong auth mode is the #1 onboarding error.
## Client setup
**One-click install (recommended):**
![Add Olostep MCP server to Cursor]()
Replace `YOUR_API_KEY` in the resulting config with your real key.
**Manual setup:**
Create or edit `.cursor/mcp.json` in your project root (or `~/.cursor/mcp.json` for global):
```json theme={null}
{
"mcpServers": {
"olostep": {
"url": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
```json theme={null}
{
"mcpServers": {
"olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY"
}
}
}
}
```
Requires Node.js 18+ on your machine.
**Verify:** Open Cursor → Settings → MCP. You should see `olostep` listed with **10 tools** including `scrape_website`. If you see "Connected, 0 tools", your API key is wrong.
**CLI install (recommended):**
```bash theme={null}
claude mcp add --transport http olostep https://mcp.olostep.com/mcp \
--header "Authorization: Bearer YOUR_API_KEY"
```
**Manual setup:**
Add to your Claude Code MCP config (`.mcp.json` in project root, or `~/.claude.json` globally):
```json theme={null}
{
"mcpServers": {
"olostep": {
"url": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
```bash theme={null}
claude mcp add --transport stdio --env OLOSTEP_API_KEY=YOUR_API_KEY olostep \
-- npx -y olostep-mcp
```
Or as JSON:
```json theme={null}
{
"mcpServers": {
"olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY"
}
}
}
}
```
**Verify:** Run `/mcp` in Claude Code. You should see `olostep` connected with 10 tools.
**Config file location:**
| OS | Path |
| ------- | ----------------------------------------------------------------- |
| macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` |
| Windows | `%APPDATA%\Claude\claude_desktop_config.json` |
| Linux | `~/.config/Claude/claude_desktop_config.json` |
**Hosted (recommended):**
```json theme={null}
{
"mcpServers": {
"olostep": {
"url": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
```json theme={null}
{
"mcpServers": {
"olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY"
}
}
}
}
```
Or install via Smithery:
```bash theme={null}
npx -y @smithery/cli install @olostep/olostep-mcp-server --client claude
```
Claude Desktop must be **fully quit and relaunched** for config changes to take effect — closing the window isn't enough (it stays running in the menu bar / system tray).
**Verify:** Open Claude Desktop → look for the 🔨 (hammer) icon in the chat input. Click it — you should see 10 Olostep tools listed.
VS Code's MCP support is built into GitHub Copilot (Agent mode). Add this to `.vscode/mcp.json` in your project, or your user `settings.json`:
```json theme={null}
{
"servers": {
"olostep": {
"type": "http",
"url": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
```json theme={null}
{
"servers": {
"olostep": {
"type": "stdio",
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY"
}
}
}
}
```
**Verify:** Open the Copilot chat panel → switch to Agent mode → the tools popover should list Olostep tools.
Add to `~/.codeium/windsurf/mcp_config.json`:
```json theme={null}
{
"mcpServers": {
"olostep": {
"serverUrl": "https://mcp.olostep.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
```json theme={null}
{
"mcpServers": {
"olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY"
}
}
}
}
```
**Verify:** Cascade → Settings → MCP. `olostep` should appear with 10 tools.
If you'd rather run the server in a container (CI, isolated env, no Node on host):
```bash theme={null}
docker pull olostep/mcp-server
docker run -i --rm \
-e OLOSTEP_API_KEY="YOUR_API_KEY" \
olostep/mcp-server
```
In an MCP client config (stdio):
```json theme={null}
{
"mcpServers": {
"olostep": {
"command": "docker",
"args": [
"run", "-i", "--rm",
"-e", "OLOSTEP_API_KEY=YOUR_API_KEY",
"olostep/mcp-server"
]
}
}
}
```
Supports `linux/amd64` and `linux/arm64`. Source on [GitHub](https://github.com/olostep/olostep-mcp-server).
1. Open the [Metorial dashboard](https://metorial.com)
2. Navigate to **MCP Servers**
3. Search for **Olostep**
4. Click **Install** and paste your API key
For manual configuration:
```json theme={null}
{
"olostep": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY"
}
}
}
```
## Picking the right tool
The MCP server exposes 10 tools. Use this decision tree to pick the right one — the agent uses the same reasoning:
| You want... | Use | Notes |
| ------------------------------------------ | ----------------------------------------- | --------------------------------------------- |
| A specific page's content | `scrape_website` or `get_webpage_content` | Set `wait_before_scraping=2000–5000` for SPAs |
| A natural-language web answer with sources | `answers` | Returns AI synthesis + citations |
| Search results for a query | `search_web` | Parser-based, non-AI, structured |
| A list of URLs on a site | `create_map` | URL discovery only — does NOT scrape |
| URLs filtered by query | `get_website_urls` | Ranked by relevance to your `search_query` |
| Many known URLs at once | `batch_scrape_urls` + `get_batch_results` | Async — kicks off, then poll |
| A whole site or section | `create_crawl` + `get_crawl_results` | Async — follows links from a start URL |
**Scraping a whole site?** Use `create_crawl`, not `batch_scrape_urls`. Crawl discovers AND scrapes. Batch is for a known list of URLs you already have.
### Tool details
Extract content from a single URL. Supports `markdown`, `html`, `json`, `text`. Optional `country` for geo-targeted requests, `wait_before_scraping` (0–10000 ms) for JS-heavy sites, and `parser` (e.g. `@olostep/amazon-product`) for structured extraction.
Lightweight markdown-only version of `scrape_website`. Use when you just want clean markdown and don't need format options.
Structured (parser-based) web search results for a query. Optional `country` for localized results. Returns JSON, not AI prose.
AI-powered answer to a `task` with sources and citations. Pass a `json` argument to get the answer in a specific shape — either a JSON schema or a short natural-language description.
Async scrape of 2–10k URLs you already have. Returns a `batch_id` — then call `get_batch_results` to fetch content. Set `wait_for_completion_seconds` (up to 900) if you want a single blocking call instead of polling. Recommended: 60 for batches under 50 URLs, 300–600 for 50–1k, 0 (poll separately) for larger batches.
Fetches the status and scraped content for a `batch_id`. Returns `processing` until done, then `completed` with the items array.
Async crawl that follows links from a `start_url`. Use `include_url_patterns` / `exclude_url_patterns` (glob syntax like `/blog/**`) to scope. Returns a `crawl_id` — then call `get_crawl_results`.
Fetches the status and pages for a `crawl_id`. Supports pagination via `cursor` and `items_limit` (max 100 per call). Returns `in_progress` until done.
Get a list of URLs on a site. URL discovery only — does not scrape. Use when you want to surface candidate URLs (e.g. let the user pick a subset). Supports `include_url_patterns` / `exclude_url_patterns` and `search_query`.
Like `create_map`, but URLs are ranked by relevance to a required `search_query`. Use when you want the top N matching links on a site.
## Troubleshooting
Your API key is invalid or rate-limited. Open the [API keys dashboard](https://www.olostep.com/dashboard/api-keys) and verify the key. If using the hosted endpoint, the header must be **exactly** `Authorization: Bearer sk_...` — no quotes around the value, no extra spaces.
Node.js isn't installed (or not in your PATH). Install Node 18+ from [nodejs.org](https://nodejs.org/), then restart your terminal **and** your MCP client. On Windows, switch to a CMD/PowerShell that has Node on the PATH.
You're likely behind a corporate proxy or firewall blocking the host. Switch to the local stdio install (`npx -y olostep-mcp`) — it makes outbound requests to `api.olostep.com` instead, which is usually allowed.
The client cached the old config. Fully quit and relaunch — not just close the window. Claude Desktop in particular keeps running in the menu bar / system tray.
If `npx` errors out launching the server on Windows, use the CMD-wrapped form:
```json theme={null}
{
"command": "cmd",
"args": ["/c", "npx", "-y", "olostep-mcp"],
"env": { "OLOSTEP_API_KEY": "YOUR_API_KEY" }
}
```
You hit the hosted endpoint without an auth header (or with the wrong format). Add the header to your client config exactly as shown in the setup tab.
## Recipes
Copy-paste prompts that work well with the tools:
* **Scrape a list of product URLs:** *"I have a CSV of 200 Amazon product URLs. Batch scrape them with `parser=@olostep/amazon-product` and return as JSON."*
* **Crawl a docs site:** *"Crawl [https://stripe.com/docs](https://stripe.com/docs) with `max_pages=50` and `include_url_patterns=['/docs/**']`. Summarize each section as markdown."*
* **Find competitors:** *"Use `answers` to find the top 5 competitors to Notion for technical doc sites. Return name, homepage, and 1-line positioning."*
* **Map then scrape:** *"Run `create_map` on [https://example.com](https://example.com) filtered to `/blog/**`, then `batch_scrape_urls` on the top 20 results."*
## Source & versions
* [GitHub repo](https://github.com/olostep/olostep-mcp-server)
* [npm package](https://www.npmjs.com/package/olostep-mcp)
* [Docker Hub](https://hub.docker.com/r/olostep/mcp-server)
* [MCP Registry](https://registry.modelcontextprotocol.io/)
# Olostep + n8n
Source: https://docs.olostep.com/integrations/n8n
Add web scraping, search, batch jobs, crawls, and site maps to any n8n workflow, no code required.
The verified [Olostep Web Scraper node](https://n8n.io/integrations/olostep-web-scraper/) gives you six operations inside n8n's visual builder: scrape a URL, search the web, get AI answers, batch-scrape thousands of URLs, crawl a site, or map all its links.
[View on n8n →](https://n8n.io/integrations/olostep-web-scraper/)
## Before you start
* **An Olostep account with an API key:** [get one free](https://olostep.com/dashboard), no credit card required. Your first 500 credits are included.
* **n8n running:** either [n8n Cloud](https://n8n.io/cloud/) or a self-hosted instance. Community nodes must be enabled (they are by default on most setups).
* **No coding required:** everything in this guide is done through n8n's visual editor.
***
## Setup
Open any workflow, click **+**, and search for **Olostep**. Select **Olostep Web Scraper** from the results.
![Search for Olostep in the n8n node picker]()
Click the result to open the Node details panel, then click **Install node**. n8n will install `n8n-nodes-olostep` and prompt you to restart. Do that before continuing.
![Olostep Web Scraper node details with Install node button]()
If **Community Nodes** is disabled for your workspace, an admin needs to enable it first. See the [n8n community nodes guide](https://docs.n8n.io/integrations/community-nodes/installation/).
Open the Olostep node in your workflow, click **Set up Credential** (in the Parameters tab), add your API key, and click **Save**.
![Olostep credentials form in n8n with API Key field]()
Get your key from the [Olostep dashboard →](https://olostep.com/dashboard)
Connect the Olostep node to a trigger and any downstream steps, then execute your workflow.
![n8n workflow canvas with Schedule Trigger connected to Olostep node]()
***
## Actions
Pull content from any URL as Markdown, HTML, JSON, or plain text. Handles JS-rendered pages with optional wait times and country targeting.
Run a web search and get structured results (titles, URLs, and snippets) as JSON.
Ask a natural-language question and get an answer with cited sources. Useful before LLM nodes when you need grounded responses.
Submit up to 10,000 URLs in one job, processed in parallel. Returns a `batch_id`; retrieve results asynchronously.
Start from a URL, follow links, and scrape all subpages. Good for docs sites, blogs, or full-site ingestion. Returns a `crawl_id`.
Get every URL on a site without scraping content. Use it for discovery before a batch job. Returns a `map_id`.
**Batch, Crawl, and Map are async.** Store the returned ID and use a Wait node or a second workflow to retrieve results once processing completes.
***
## Example workflow: Lead enrichment from Google Sheets
**What it does:** When you paste a company URL into a Google Sheet, this workflow automatically scrapes the company's website, extracts key information with an AI node, and writes the results back to the same row, turning a blank spreadsheet into a filled-out lead database.
**Nodes used:** Google Sheets trigger → Olostep Scrape Website → OpenAI → Code → Google Sheets update
![Lead enrichment workflow in n8n: Google Sheets trigger connected to Olostep, OpenAI, Code, and Google Sheets update nodes]()
***
### Step 1: Set up your Google Sheet
Create a sheet with these columns: `Company URL`, `Industry`, `Description`, `Company Size`, `Enriched`. The workflow reads from `Company URL` and fills in the rest.
### Step 2: Add a Google Sheets trigger
In n8n, add a **Google Sheets** trigger node. Set the event to **Row Added**, point it at your sheet, and set it to watch the `Company URL` column. Now every time you paste a new URL into the sheet, this workflow fires.
### Step 3: Add Olostep Scrape Website
Connect an **Olostep Web Scraper** node after the trigger. Set:
* **Action:** Scrape Website
* **URL:** `{{ $json["Company URL"] }}` (pulls the URL from the new row)
* **Output Format:** Markdown
Markdown works best here because it strips navigation, ads, and boilerplate. The AI node in the next step gets clean prose about the company instead of raw HTML noise.
### Step 4: Add an OpenAI node
Connect an **OpenAI** node. Set the model to `gpt-4o-mini` (fast and cheap for extraction tasks) and use this prompt:
```
You are a sales researcher. Based on the company website content below, extract:
1. Industry (one phrase, e.g. "B2B SaaS", "E-commerce", "Healthcare")
2. One-sentence company description (max 20 words)
3. Estimated company size (Startup / SMB / Mid-market / Enterprise)
Return only a JSON object with keys: industry, description, company_size.
Website content:
{{ $json.markdownContent }}
```
The `markdownContent` field is what Olostep returns from the scrape, as clean plain text.
### Step 5: Parse the AI response and write back
Add a **Code** node to parse the JSON from OpenAI:
```js theme={null}
const parsed = JSON.parse($input.first().json.message.content);
return [{ json: parsed }];
```
Then connect a **Google Sheets** node set to **Update Row**. Map the columns:
* `Industry` → `{{ $json.industry }}`
* `Description` → `{{ $json.description }}`
* `Company Size` → `{{ $json.company_size }}`
* `Enriched` → `Yes`
### What you get
Paste a URL like `https://notion.so` into your sheet, and within \~10 seconds the row fills in:
| Company URL | Industry | Description | Company Size | Enriched |
| -------------------------------------- | ----------------- | --------------------------------------------------- | ------------ | -------- |
| [https://notion.so](https://notion.so) | Productivity SaaS | All-in-one workspace for notes, docs, and databases | Mid-market | Yes |
From here you can extend this workflow: add a Slack notification when enrichment completes, filter by industry before writing back, or replace Google Sheets with HubSpot to update contacts directly.
***
## Templates
Ready-to-import n8n workflows built with Olostep:
Crawl documentation sites with Olostep and structure the output into an AI-ready knowledge base.
Scrape business leads from Google Maps and enrich them with decision-maker details.
Analyze complaints with Olostep + Gemini and generate structured insight reports in Google Docs.
Extract Amazon product URLs and metadata with Olostep, then sync the results to Sheets.
[Browse all Olostep workflows on n8n.io →](https://n8n.io/workflows/?q=olostep)
***
## Parsers
Add a parser ID to the **Parser** field on any Scrape or Batch action to get structured data instead of raw content:
| Parser | Extracts |
| ---------------------------- | ------------------------------------------------ |
| `@olostep/amazon-product` | Title, price, rating, reviews, images, variants |
| `@olostep/google-search` | Result titles, URLs, snippets |
| `@olostep/google-maps` | Business name, address, rating, reviews |
| `@olostep/extract-emails` | Email addresses from any page |
| `@olostep/extract-socials` | Social profile links (X, GitHub, LinkedIn, etc.) |
| `@olostep/extract-calendars` | Google Calendar and ICS links |
See the full list in the [Olostep parser store →](https://www.olostep.com/store)
***
## Troubleshooting
Copy the key directly from [olostep.com/dashboard](https://olostep.com/dashboard) with no trailing spaces. Delete and recreate the credential in n8n if the error persists.
Increase **Wait Before Scraping** (try 2000–5000ms for JS-heavy pages). Confirm the URL is publicly accessible without a login. If a specific domain is consistently failing, contact [info@olostep.com](mailto:info@olostep.com).
The **URLs to Scrape** field expects a JSON array:
```json theme={null}
[
{ "url": "https://example.com/page-1", "custom_id": "p1" },
{ "url": "https://example.com/page-2", "custom_id": "p2" }
]
```
Use a Code node upstream to build this array from your data if needed.
Add a **Wait** node between scrape steps, or switch to **Batch Scrape URLs** instead of looping single scrapes. Check current usage in the [dashboard](https://olostep.com/dashboard).
On n8n Cloud, community nodes must be enabled by a workspace owner. On self-hosted, make sure `N8N_COMMUNITY_PACKAGES_ENABLED=true` is set in your environment. See [n8n's installation guide](https://docs.n8n.io/integrations/community-nodes/installation/).
***
## Related
Full reference for the scrape endpoint
How batch jobs work and how to retrieve results
Crawl configuration and result retrieval
URL discovery and filtering options
## Get Started
Ready to automate your web search, scraping, and crawling workflows?
n8n platform
Install n8n-nodes-olostep and start building automated workflows
Connect Olostep with n8n and automate your web data extraction today!
# Olostep + Nanobot Integration
Source: https://docs.olostep.com/integrations/nanobot
Use Olostep as a web-search backend for nanobot's web_search tool.
Olostep adds a search backend for nanobot's `web_search` tool, giving agents AI-friendly web answers and source links without requiring you to build a custom retrieval pipeline.
## Features
Return a concise answer plus supporting source links.
Enable the provider with a single config value and an API key.
Install Olostep only when you need it.
Route requests through `tools.web.proxy` when required.
Falls back to DuckDuckGo when no Olostep key is available.
Uses the same web-search output formatting as the other providers.
## Installation
```bash pip theme={null}
pip install "nanobot-ai[olostep]"
```
```bash poetry theme={null}
poetry add "nanobot-ai[olostep]"
```
If you manage dependencies manually, the underlying package is `olostep>=0.1.0`.
## Setup
Set your API key with either an environment variable or your nanobot config.
### Environment variable
```bash theme={null}
export OLOSTEP_API_KEY="your-api-key"
```
### Config file
Add this to `~/.nanobot/config.json`:
```json theme={null}
{
"tools": {
"web": {
"search": {
"provider": "olostep",
"apiKey": "YOUR_OLOSTEP_API_KEY"
}
}
}
}
```
### Optional proxy
If your environment requires a proxy, configure it once under `tools.web.proxy`:
```json theme={null}
{
"tools": {
"web": {
"proxy": "http://127.0.0.1:7890"
}
}
}
```
## Available Tools / Methods
### `web_search`
Use Olostep by setting `tools.web.search.provider` to `olostep`.
#### Parameters
Set to `olostep` to enable this integration. Default: `duckduckgo`
Olostep API key. You can also use `OLOSTEP_API_KEY` environment variable.
Not used by Olostep. Kept for config consistency.
Results per search, from 1–10.
Search timeout in seconds.
Proxy URL configured under `tools.web`.
```json Basic Setup theme={null}
{
"tools": {
"web": {
"search": {
"provider": "olostep",
"apiKey": "YOUR_OLOSTEP_API_KEY"
}
}
}
}
```
```json With Environment Variable theme={null}
{
"tools": {
"web": {
"search": {
"provider": "olostep"
}
}
}
}
```
```bash theme={null}
export OLOSTEP_API_KEY="your-api-key"
```
```json With Proxy theme={null}
{
"tools": {
"web": {
"proxy": "http://127.0.0.1:7890",
"search": {
"provider": "olostep",
"apiKey": "YOUR_OLOSTEP_API_KEY"
}
}
}
}
```
## Full Agent Examples
### Example 1: Quick Research Assistant
```python theme={null}
import asyncio
from nanobot import Nanobot
async def main() -> None:
bot = Nanobot.from_config()
result = await bot.run(
"Use web search to summarize the latest Olostep SDK capabilities and cite sources.",
session_key="olostep-research",
)
print(result.content)
asyncio.run(main())
```
### Example 2: Research Workflow in a Workspace
```python theme={null}
import asyncio
from nanobot import Nanobot
async def main() -> None:
bot = Nanobot.from_config(workspace="/home/user/projects/research")
result = await bot.run(
"Find recent documentation for programmatic web access tools and list the key tradeoffs.",
session_key="olostep-workspace-demo",
)
print(result.content)
asyncio.run(main())
```
### Example 3: Proxy-Aware Search Setup
```python theme={null}
import asyncio
from nanobot import Nanobot
async def main() -> None:
bot = Nanobot.from_config(workspace="/home/user/projects/research")
result = await bot.run(
"Search for implementation notes about web-scraping SDKs and summarize the differences.",
session_key="olostep-proxy-demo",
)
print(result.content)
asyncio.run(main())
```
## Configuration / Options
* Set `tools.web.search.provider` to `olostep` to enable the integration.
* Keep `tools.web.enable` as `true` if you want both `web_search` and `web_fetch`.
* Set `tools.web.enable` to `false` to disable all built-in web tools.
* Set `tools.web.proxy` if your environment requires outbound traffic through a proxy.
* Leave `provider` unset if you want the default DuckDuckGo fallback behavior.
### Fallback behavior
If Olostep is selected but no API key is available, nanobot falls back to DuckDuckGo instead of failing hard.
## Specialized Features
* **Source-aware answers** — Olostep returns a response plus source links.
* **Shared formatting** — results are rendered in the same normalized search output used by the other providers.
* **No hard dependency** — the provider is imported behind a `try/except` guard, so nanobot still works without Olostep installed.
* **Proxy-aware transport** — `tools.web.proxy` is applied to the underlying HTTP client used by the integration.
## Pricing
Olostep pricing is managed by Olostep directly and may change over time. Check your Olostep account dashboard for current plans, quotas, and usage costs.
## Support
* **PyPI**: [pypi.org/project/olostep](https://pypi.org/project/olostep/)
* **Documentation**: [docs.olostep.com](https://docs.olostep.com)
* **Homepage**: [olostep.com](https://www.olostep.com)
* **GitHub repository**: [github.com/olostep-api/olostep-py](https://github.com/olostep-api/olostep-py)
* **GitHub issues**: [github.com/olostep-api/olostep-py/issues](https://github.com/olostep-api/olostep-py/issues)
* **Email**: [team@olostep.com](mailto:team@olostep.com)
## Related Resources
Learn about the Answers endpoint powering this integration
Explore the Olostep Python SDK
Understand web search capabilities
Browse the full API reference
# Olostep + Raycast Integration
Source: https://docs.olostep.com/integrations/raycast
Add web scraping, search, crawling, and AI answers to Raycast AI with the official Olostep MCP server
Raycast is the productivity launcher used by developers, operators, and AI power users. With Raycast's native support for the Model Context Protocol (MCP), the **official Olostep MCP server** plugs straight into Raycast AI — giving your launcher live web data tools you can call from AI Chat, Quick AI, and AI Commands with a simple `@olostep` mention.
Olostep is an [official entry in Raycast's MCP registry](https://www.raycast.com/), so you can add it in a couple of clicks — no config files to edit.
## Features
The Olostep MCP Server gives your Raycast AI access to Olostep's core web data capabilities:
Extract content from any single URL as Markdown, HTML, JSON, or text — with JavaScript rendering and country targeting.
Run a web search and get structured, parser-based results (titles, URLs, snippets).
Ask a natural-language question and get a web-grounded answer with cited sources and optional structured output.
Process up to 10,000 URLs in a single parallel job — ideal for large-scale extraction.
Start from a URL, follow links, and scrape entire sites or sections automatically.
Discover every URL on a site without scraping — perfect for planning a crawl or batch job.
## Before you start
* **An Olostep API key:** [get one free](https://olostep.com/dashboard/api-keys), no credit card required. Your first 500 credits are included.
* **Raycast installed**, with [Raycast AI](https://manual.raycast.com/ai) available on your plan. MCP tools run through Raycast AI (AI Chat, Quick AI, and AI Commands).
Olostep supports two connection modes. The **hosted endpoint** (`https://mcp.olostep.com/mcp`) uses an `Authorization: Bearer YOUR_API_KEY` header. The **local stdio** option (`npx -y olostep-mcp`) uses `OLOSTEP_API_KEY` as an environment variable. Pick one — mixing the two auth modes is the most common setup mistake.
## Installation
Copy your key from the [Olostep dashboard](https://olostep.com/dashboard/api-keys). You'll paste it into Raycast in the next step.
Because Olostep is an official entry in Raycast's MCP registry, the fastest path is one-click:
1. Install the **Model Context Protocol Registry** extension from the Raycast Store (search "MCP" or "Model Context Protocol").
2. Run the **Search Servers** command, search for **Olostep**, and select it.
3. Choose **Install**, then enter your Olostep API key when prompted.
Raycast saves the configuration, starts the connection, and loads the tools automatically.
Prefer to set it up yourself? Open Raycast, run **Install MCP Server** (or **Manage MCP Servers → Install New Server**), and fill out the form with these values:
* **Name:** `olostep`
* **Transport:** HTTP
* **URL:** `https://mcp.olostep.com/mcp`
* **HTTP Header:** `Authorization` → `Bearer YOUR_API_KEY`
Press **Install MCP Server** (`⌘` / `Ctrl` + `↵`).
Choose **Standard Input/Output** as the transport instead, and use:
* **Command:** `npx`
* **Arguments:** `-y olostep-mcp`
* **Environment:** `OLOSTEP_API_KEY` → `YOUR_API_KEY`
Requires Node.js 18+ on your machine. If the command relies on something on your `PATH`, restart Raycast after updating your environment variables so the new values are picked up.
Open the **Manage MCP Servers** command — `olostep` should appear as **Running** with its **10 tools**. Then, in AI Chat or Quick AI, type `@olostep` to scope a request to it:
```txt theme={null}
@olostep find the latest pricing on these three competitor pages and summarize it
```
## Available tools
In Raycast you invoke tools in natural language — `@`-mention `olostep`, describe what you want, and Raycast AI picks the right tool. The examples below show typical prompts.
### scrape\_website
Extract content from a single URL in Markdown, HTML, JSON, or text. Handles JavaScript-rendered pages and supports country targeting and specialized parsers.
```txt theme={null}
@olostep scrape https://example.com/pricing and give me the clean markdown
```
### search\_web
Run a web search and return structured results for a query.
```txt theme={null}
@olostep search for the latest news on EU AI regulation
```
### answers
Get an AI-generated answer to a question, grounded in real web data and returned with cited sources. Ask for a specific shape to get structured output.
```txt theme={null}
@olostep what are the top 5 competitors to Notion for technical docs? Return name, homepage, and a one-line description, with sources.
```
### batch\_scrape\_urls
Scrape a known list of URLs in one parallel job (up to 10,000). Runs asynchronously and returns a job you retrieve once it completes.
```txt theme={null}
@olostep scrape these 200 product URLs and return the title and price for each as JSON
```
### create\_crawl
Start from a URL, follow its links, and scrape matching pages across a whole site or section. Runs asynchronously.
```txt theme={null}
@olostep crawl https://docs.stripe.com under /docs and summarize each section as markdown
```
### create\_map
List every URL on a site without scraping content — useful for discovery before a crawl or batch job.
```txt theme={null}
@olostep map all the URLs under https://example.com/blog
```
**Batch and crawl run asynchronously.** Each kicks off a job and returns an ID, which Raycast retrieves with its companion tool (`get_batch_results`, `get_crawl_results`). Together with `get_webpage_content` and `get_website_urls`, the server exposes **10 tools** in total. See the [Olostep MCP Server reference](/integrations/mcp-server) for the full list and parameters.
## Example workflows
### Quick competitor research
From AI Chat, gather and synthesize web data in one prompt:
```txt theme={null}
@olostep search for the top project management tools in 2026, then use answers
to summarize the 3 most-mentioned ones with their pricing and a source for each.
```
### Turn a page into clean notes
```txt theme={null}
@olostep scrape https://example.com/blog/post and give me a 5-bullet summary
plus the key quotes.
```
### Save it as a reusable AI Command
Once Olostep is connected, you can bake a prompt into a one-press [AI Command](https://manual.raycast.com/ai). For example, create a command named **"Summarize URL"** with the instruction:
```txt theme={null}
Using @olostep, scrape {argument} as markdown and return a 3-paragraph summary
covering what the page is about, the key details, and any pricing.
```
Now you can run it from anywhere in Raycast with a URL and get an instant summary.
## Troubleshooting
Your API key is invalid or rate-limited. Verify it in the [Olostep dashboard](https://olostep.com/dashboard/api-keys). For the hosted endpoint, the header must be exactly `Authorization: Bearer YOUR_API_KEY` — no quotes around the value and no extra spaces.
Open **Manage MCP Servers** and check the server's status. If it's in an **Error** state, the details pane shows the full output from the server or transport. Re-check the URL/header (HTTP) or command/env (stdio), then use the **Restart** action.
Make sure Node.js 18+ is installed and on your `PATH`. After installing or changing environment variables, fully restart Raycast so the new values are picked up. On Windows, ensure `npx` resolves in the shell Raycast launches.
Confirm `https://mcp.olostep.com/mcp` is reachable from your network (corporate proxies and firewalls can block it). If it's blocked, switch to the local stdio install, which connects to `api.olostep.com` instead.
Make sure you `@`-mention `olostep` in the message (in AI Chat or Quick AI), and that the request clearly calls for web data. You can scope a question to the server by typing `@` and selecting it.
## Learn more
The Raycast launcher
How Raycast connects and manages MCP servers
Full MCP setup, tool reference, and parameters
Source, issues, and release notes
## Related
Full reference for the scrape endpoint
How batch jobs work and how to retrieve results
Crawl configuration and result retrieval
URL discovery and filtering options
## Get started
Ready to bring live web data into Raycast?
Sign up and grab your Olostep API key — 500 credits included
Find Olostep and other servers in the Raycast MCP registry
Connect Olostep with Raycast and put web scraping, search, and crawling one `@olostep` mention away.
# Olostep + Relay
Source: https://docs.olostep.com/integrations/relay
Add web scraping, AI-powered research, batch jobs, crawls, and site maps to any Relay workflow, no code required.
The verified [Olostep app on Relay](https://www.relay.app/apps/olostep/integrations) gives you two actions inside Relay's visual builder: scrape a URL or map all links on a site.
[View on Relay →](https://www.relay.app/apps/olostep/integrations)
## Before you start
* **An Olostep account with an API key:** [get one free](https://olostep.com/dashboard), no credit card required. Your first 500 credits are included.
* **A Relay account:** create one at [relay.app](https://www.relay.app/).
* **No coding required:** everything in this guide is done through Relay's visual editor.
***
## Setup
Open a workflow in Relay, add a step, and search for **Olostep**. Select **Olostep** from the app list.
![Placeholder: Search for Olostep in Relay app picker]()
After selecting **Olostep**, choose one of the available Olostep actions in the step configuration.
![Placeholder: Olostep action selector in Relay]()
Click **Connect account**, paste your Olostep API key, and authorize the connection. Relay will save this account for future workflows.
![Placeholder: Relay account connection modal with Olostep API key field]()
Get your key from the [Olostep dashboard →](https://olostep.com/dashboard)
* **URL to Scrape:** map from a previous step or enter manually
* **Output Format:** choose `Markdown`, `HTML`, `JSON`, or `Text`
![Placeholder: Relay Olostep Scrape Website fields with URL and output format]()
Run a test to verify output, then publish or turn on the workflow once results look correct.
![Placeholder: Relay test output and publish flow]()
***
## Actions
Pull content from any URL as Markdown, HTML, JSON, or plain text. Handles JS-rendered pages with optional wait times and country targeting.
Get every URL on a site without scraping content. Use it for discovery before a batch job. Returns a `map_id`.
**Map is async.** Store the returned `map_id` and use a delay/poll pattern in Relay to retrieve results once processing completes.
***
## Example workflow: Scheduled competitor page scrape
**What it does:** On a schedule, this workflow scrapes a competitor page and stores clean content you can reuse in downstream Relay steps.
**Nodes used:** Schedule -> Olostep Scrape Website
![Placeholder: Relay canvas showing schedule and Olostep scrape steps]()
***
### Step 1: Add a schedule trigger
Create a workflow and add a schedule trigger (for example, every weekday at 8 AM).
### Step 2: Add Olostep Scrape Website
Add an **Olostep** step and select **Scrape Website**. Set:
* **URL:** `https://competitor.com/blog`
* **Output Format:** Markdown
### Step 3: Test and publish
Run a test to confirm the output, then publish or turn on the workflow.
### What you get
Every run produces clean page content ready for analysis or routing:
> **Scrape Result**
>
> * Source URL
> * Retrieved content in your selected format
> * Timestamped run output in Relay
***
## Parsers
Add a parser ID to the **Parser** field on the Scrape action to get structured data instead of raw content:
| Parser | Extracts |
| ---------------------------- | ------------------------------------------------ |
| `@olostep/amazon-product` | Title, price, rating, reviews, images, variants |
| `@olostep/google-search` | Result titles, URLs, snippets |
| `@olostep/google-maps` | Business name, address, rating, reviews |
| `@olostep/extract-emails` | Email addresses from any page |
| `@olostep/extract-socials` | Social profile links (X, GitHub, LinkedIn, etc.) |
| `@olostep/extract-calendars` | Google Calendar and ICS links |
See the full list in the [Olostep parser store →](https://www.olostep.com/store)
***
## Relay.app vs Zapier
Relay.app is a strong alternative to Zapier for workflows that need built-in review and AI-first orchestration.
**Relay.app advantages:**
* **Human-in-the-loop:** native approval and review steps in the workflow
* **AI-first workflow design:** easier to add AI decisions and summaries as first-class steps
* **Workflow clarity:** clean visual builder with clear run context and step outputs
**When Zapier may be better:**
* You need coverage for a niche app only available on Zapier
* Your team already has many existing Zaps and operational tooling around Zapier
If both platforms support your stack, choose Relay when review + AI collaboration are central to your process.
***
## Troubleshooting
Copy the key directly from [olostep.com/dashboard](https://olostep.com/dashboard) with no trailing spaces. Reconnect the Olostep account in Relay if the error persists.
Increase **Wait Before Scraping** (try 2000–5000ms for JS-heavy pages). Confirm the URL is publicly accessible without login. If one domain consistently fails, contact [info@olostep.com](mailto:info@olostep.com).
Add delays between scrape-heavy runs and retry with backoff for large workflows. Check current usage in the [dashboard](https://olostep.com/dashboard).
Maps are async. Store the returned `map_id` first, then fetch results in a later step/run once processing is complete.
***
## Related
Full reference for the scrape endpoint
How batch jobs work and how to retrieve results
Crawl configuration and result retrieval
URL discovery and filtering options
# Olostep + ViaSocket
Source: https://docs.olostep.com/integrations/viasocket
Connect Olostep to any AI client — Claude, Cursor, ChatGPT, and more — using ViaSocket Mushrooms as an MCP gateway.
[ViaSocket Mushrooms](https://mushrooms.viasocket.com/) is an MCP gateway that bridges apps like Olostep to any AI client through a single MCP endpoint URL. Once you connect Olostep as a Mushroom, you can instruct Claude, Cursor, ChatGPT, or any other supported client to scrape websites, run batch jobs, crawl sites, and get AI-grounded answers — all in plain language, without writing a single line of code.
## Before you start
* **An Olostep account with an API key:** [get one free](https://olostep.com/dashboard), no credit card required. Your first 500 credits are included.
* **A ViaSocket account:** create one at [viasocket.com](https://viasocket.com/).
* **An AI client:** Claude, Cursor, ChatGPT, Windsurf, VS Code, or any MCP-compatible client.
***
## Setup
Open [ViaSocket Mushrooms](https://mushrooms.viasocket.com/) and click **+ New Cluster**. ViaSocket will prompt you to choose the AI client this Cluster will connect to — select Claude, Cursor, ChatGPT, or whichever client you use.
![ViaSocket New Cluster dialog — Choose your AI client, showing Claude, ChatGPT, Cursor, Windsurf and more]()
Inside your Cluster, click **+ BROWSE INTEGRATIONS** to open the Mushrooms library — a catalog of 2,500+ apps you can give your AI access to.
![ViaSocket Mushrooms dashboard showing Add New Mushroom and Browse Integrations button]()
Search for **olostep** in the search bar. Select **Olostep** from the results.
![Searching for olostep in ViaSocket Mushrooms — Olostep appears in results]()
On the Olostep detail page, click **CONNECT TO OLOSTEP** to authorize your account and start adding Olostep actions to your Cluster.
![Olostep detail page in ViaSocket showing Connect to get started and CONNECT TO OLOSTEP button]()
Choose which Olostep actions your AI client can use. All seven are enabled by default — toggle off any you don't need, then click **NEXT**.
![Olostep Select Actions screen in ViaSocket showing 7 available actions with toggle switches]()
Paste your Olostep API key into the **API Key** field and click **ADD CONNECTION**. ViaSocket will verify the key and save the credential.
![Add New Connection screen for Olostep in ViaSocket with API Key input field]()
Get your key from the [Olostep dashboard →](https://olostep.com/dashboard)
Olostep now appears as an enabled Mushroom in your Cluster. Copy the **MCP Endpoint URL** (or the full JSON config block) from the **Cursor Configuration** panel and paste it into your AI client's MCP settings file.
![ViaSocket Cluster showing Olostep enabled and the MCP endpoint URL with JSON config ready to copy]()
The config looks like this — paste it into your AI client's settings:
```json theme={null}
{
"mcpServers": {
"viasocket": {
"type": "http",
"url": "https://mcp.viasocket.com/mcp/"
}
}
}
```
***
## Actions
Pull content from any URL as Markdown, HTML, JSON, or plain text. Handles JS-rendered pages with optional wait times and country targeting.
Ask a natural-language question and get a cited answer grounded in pages you provide or a live web search.
Submit up to 100,000 URLs in one job, processed in parallel. Returns a `batch_id` for async retrieval.
Retrieve the results of a completed batch job using its `batch_id`.
Start from a seed URL, follow links, and scrape all subpages. Returns a `crawl_id`.
Check the status and retrieve results for a running or completed crawl by `crawl_id`.
Get every URL on a site without scraping content. Useful for discovery before a batch job.
***
## Example: Ask Claude to research a competitor
Once your MCP endpoint is configured in Claude, you can issue plain-language instructions like:
> *"Scrape the pricing page at acme.com/pricing and give me a summary of their plans and prices."*
Claude will call Olostep's **Scrape Website** action through ViaSocket, return the page content, and summarize it — no workflow builder, no code.
You can chain actions the same way:
> *"Map all the pages on docs.example.com, then batch-scrape the first 50 and build a knowledge base summary."*
***
## Troubleshooting
Copy the key directly from [olostep.com/dashboard](https://olostep.com/dashboard) with no trailing spaces. Delete and recreate the Olostep connection in ViaSocket if the error persists.
Make sure you've pasted the MCP endpoint URL (or JSON config block) into the correct settings file for your AI client. For Claude Desktop, this is `claude_desktop_config.json`. For Cursor, it's `.cursor/mcp.json` in your project or the global Cursor settings. Restart the client after saving.
Increase the wait time when calling **Scrape Website** (try 2000–5000ms for JS-heavy pages). Confirm the URL is publicly accessible without a login.
The endpoint URL is tied to your Cluster. If you delete and recreate the Cluster, you'll get a new URL and will need to update your AI client config. Keep the URL private — it authorizes all actions on your connected accounts.
***
## Related
Full reference for the scrape endpoint
How batch jobs work and how to retrieve results
Crawl configuration and result retrieval
URL discovery and filtering options
Use Olostep's native MCP server directly
ViaSocket platform
# Olostep + Zapier
Source: https://docs.olostep.com/integrations/zapier
Add web scraping, AI-powered research, batch jobs, crawls, and site maps to any Zap, no code required.
The verified [Olostep app on Zapier](https://zapier.com/apps/olostep/integrations) gives you five actions inside Zapier's visual builder: scrape a URL, get AI answers, batch-scrape thousands of URLs, crawl a site, or map all its links, then connect the results to any of 8,000+ apps.
[View on Zapier →](https://zapier.com/apps/olostep/integrations)
## Before you start
* **An Olostep account with an API key:** [get one free](https://olostep.com/dashboard), no credit card required. Your first 500 credits are included.
* **A Zapier account:** any plan works. Free plans can run Zaps manually; paid plans support scheduled and multi-step Zaps.
* **No coding required:** everything in this guide is done through Zapier's visual editor.
***
## Setup
Open any Zap, click **+** to add an action, and search for **Olostep**. Select **Olostep** from the results.
![Search for Olostep in the Zapier action picker]()
Click **Sign in to Olostep**. Paste your API key into the field and click **Yes, Continue to Olostep**. Zapier will verify the key and save the credential for all future Zaps.
![Olostep account connection screen in Zapier with API Key field]()
Get your key from the [Olostep dashboard →](https://olostep.com/dashboard)
With your account connected, pick one of the five Olostep actions from the **Action** dropdown and fill in the required fields.
![Olostep action dropdown in Zapier showing available actions]()
* **URL to Scrape:** map it from your trigger or enter manually
* **Output Format:** choose `Markdown`, `HTML`, `JSON`, or `Text`
![Configure Scrape Website in Zapier with URL and output format fields]()
Click **Test step** to run a live request and confirm the output. Once it looks right, connect downstream steps and publish your Zap.
![Zapier test step result and publish flow]()
***
## Actions
Pull content from any URL as Markdown, HTML, JSON, or plain text. Handles JS-rendered pages with optional wait times and country targeting.
Ask a natural-language question and get a cited answer grounded in pages you provide or a live web search.
Submit up to 100,000 URLs in one job, processed in parallel. Returns a `batch_id`; retrieve results asynchronously.
Start from a URL, follow links, and scrape all subpages. Good for docs sites, blogs, or full-site ingestion. Returns a `crawl_id`.
Get every URL on a site without scraping content. Use it for discovery before a batch job. Returns a `map_id`.
**Batch, Crawl, and Map are async.** Store the returned ID and use a Delay step or a second Zap to retrieve results once processing completes.
***
## Example workflow: Daily competitor briefing
**What it does:** Every morning, this Zap scrapes a competitor page, asks Olostep AI Answer to summarize key updates from that page, and posts the final briefing with citations to Slack.
**Nodes used:** Schedule by Zapier -> Olostep Scrape Website -> Olostep Ask AI Answer -> Slack
![Zapier canvas: Schedule trigger connected to Olostep Scrape, Olostep Ask AI Answer, and Slack]()
***
### Step 1: Add a Schedule trigger
In Zapier, create a new Zap and set the trigger to **Schedule by Zapier**. Set it to run **Every Day** at a time that works for your team (e.g. 8 AM).
### Step 2: Add Olostep Scrape Website
Add an **Olostep** action and choose **Scrape Website**. Set:
* **URL:** `https://competitor.com/blog`
* **Output Format:** Markdown
Markdown keeps the page content clean for the next AI Answer step.
### Step 3: Add Olostep Ask AI Answer
Add a second **Olostep** action and choose **Ask AI Answer**. Set:
* **Question:** `What are the key updates, announcements, product changes, or pricing changes on this page today? Return a concise briefing with bullet points and include citations.`
* **Context URLs (JSON Array):** map the **Scraped URL** from step 2
* **Format:** Markdown
* **Include Citations:** true
### Step 4: Send to Slack
Add a **Slack** action set to **Send Channel Message**. Map:
* **Channel:** `#competitive-intel`
* **Message Text:** `Daily Competitor Briefing`
* **Message Body:** `{{Answer (Markdown)}}`
### What you get
Every morning your Slack channel receives a message like:
> **Daily Competitor Briefing**
>
> * Launched a new integration for ecommerce workflows
> * Updated pricing page with a new mid-tier plan
> * Published two new blog posts about automation best practices
>
> *Sources: competitor.com/blog, competitor.com/pricing*
From here you can extend this Zap: add a Filter to only post if specific keywords appear, write briefings to Google Sheets for tracking, or run separate Zaps for different competitor URLs.
***
## Parsers
Add a parser ID to the **Parser** field on any Scrape or Batch action to get structured data instead of raw content:
| Parser | Extracts |
| ---------------------------- | ------------------------------------------------ |
| `@olostep/amazon-product` | Title, price, rating, reviews, images, variants |
| `@olostep/google-search` | Result titles, URLs, snippets |
| `@olostep/google-maps` | Business name, address, rating, reviews |
| `@olostep/extract-emails` | Email addresses from any page |
| `@olostep/extract-socials` | Social profile links (X, GitHub, LinkedIn, etc.) |
| `@olostep/extract-calendars` | Google Calendar and ICS links |
See the full list in the [Olostep parser store →](https://www.olostep.com/store)
***
## Zapier Limitations & Workarounds
### Task limits
Zapier counts each action as one task against your plan's limit.
**Workaround:** Use **Batch Scrape URLs** to process multiple URLs as a single task instead of looping a single-scrape action.
### Execution timeout
Zaps timeout after 30 seconds. Crawls and large batch jobs take longer than that.
**Workaround:** Store the returned `crawl_id` or `batch_id` and retrieve results in a separate Zap triggered by a webhook or a scheduled delay.
### Data size limits
Zapier caps the data size that can pass between steps, which can be an issue with large scrape payloads.
**Workaround:** Use hosted output URLs returned by Olostep to fetch large content separately rather than passing raw content between steps.
### Polling triggers
Most Zapier triggers poll on a 5–15 minute interval, not instantly.
**Workaround:** Use Zapier's **Webhooks** trigger for instant notification, or schedule Zaps at fixed times rather than relying on near-real-time polling.
***
## Troubleshooting
Copy the key directly from [olostep.com/dashboard](https://olostep.com/dashboard) with no trailing spaces. Disconnect and reconnect the Olostep account in Zapier if the error persists.
Increase **Wait Before Scraping** (try 2000–5000ms for JS-heavy pages). Confirm the URL is publicly accessible without a login. If a specific domain is consistently failing, contact [info@olostep.com](mailto:info@olostep.com).
The **URLs to Scrape** field expects a JSON array:
```json theme={null}
[
{ "url": "https://example.com/page-1", "custom_id": "p1" },
{ "url": "https://example.com/page-2", "custom_id": "p2" }
]
```
Use a **Code by Zapier** step upstream to build this array from your data if needed.
Add a **Delay** step between scrape actions, or switch to **Batch Scrape URLs** instead of looping single scrapes. Check current usage in the [dashboard](https://olostep.com/dashboard).
These operations are async by design. Store the returned ID immediately after the action, then use a second Zap or a scheduled poll to retrieve results later.
***
## Related
Full reference for the scrape endpoint
How batch jobs work and how to retrieve results
Crawl configuration and result retrieval
URL discovery and filtering options
Zapier platform
# Olostep CLI
Source: https://docs.olostep.com/sdks/cli
Command-line interface for scrape, search, map, crawl, answers, and batches — JSON output for scripts, CI, and AI agents
**NPM package:** [olostep-cli](https://www.npmjs.com/package/olostep-cli)
**Repository:** [github.com/olostep-api/olostep-cli](https://github.com/olostep-api/olostep-cli)
CLI for the [Olostep API](https://www.olostep.com/) — **scrape**, **search**, **map**, **crawl**, **answer**, and **batch** the web from your terminal. Every command returns **JSON** so it pipes cleanly into `jq`, agents, and CI.
Pure JavaScript, Node 18+, no native binaries to download. Installs in under a second, starts in \~200 ms, ships as a single \~100 KB bundle.
## Install
**Requirements:** Node.js **18+**.
```bash theme={null}
npm install -g olostep-cli
olostep init
```
```bash theme={null}
curl -fsSL https://olostep.com/install.sh | sh
olostep init
```
The script checks Node 18+, runs `npm install -g olostep-cli`, and falls back to `sudo` if needed.
```powershell theme={null}
iwr -useb https://olostep.com/install.ps1 | iex
olostep init
```
Same idea as the macOS/Linux script, but for PowerShell.
```bash theme={null}
npx -y olostep-cli@latest --help
```
Good for trying a single command without a global install.
`olostep init` is the recommended next step — it signs you in, installs the Olostep skills into your AI agents, and configures the MCP server, all in one command. The one-liner scripts wrap `npm install -g olostep-cli` with a Node 18+ check and a `sudo` fallback, so they work even if you're unsure of your local setup.
**Platforms:** macOS (Apple Silicon and Intel), Linux (x64 and arm64), Windows (x64 and arm64).
## Set up
One command does everything — sign in, install skills, and install the MCP server:
```bash theme={null}
olostep init
```
Flags: `--skills-only`, `--mcp-only`, `--no-browser`, `--relogin`.
To **just sign in** (no skills/MCP):
```bash theme={null}
olostep login
olostep login --no-browser # print the URL (useful over SSH)
```
The browser opens to the Olostep auth page; you click **Authorize**, and the CLI saves your key locally.
**Alternative — set an env var.** Good for CI:
```bash theme={null}
export OLOSTEP_API_KEY=your_key_here
```
Get a key from the [API Keys dashboard](https://www.olostep.com/dashboard/api-keys).
**Where the key is stored** (after `olostep login`):
| OS | Path |
| ------- | ------------------------------------------------------------ |
| macOS | `~/Library/Application Support/olostep-cli/credentials.json` |
| Linux | `~/.config/olostep-cli/credentials.json` |
| Windows | `%USERPROFILE%\AppData\Roaming\olostep-cli\credentials.json` |
## Sign out
```bash theme={null}
olostep logout # prompts to confirm, then removes credentials.json
olostep logout --dry-run # preview only — see what would happen
olostep logout --yes # skip the confirmation (for scripts)
olostep logout --json # machine-readable output
```
`logout` also warns you if `OLOSTEP_API_KEY` / `OLOSTEP_API_TOKEN` env vars or a `.env` file in your current directory still hold a key — those take priority over the credentials file, so deleting the file alone may not be enough. The output includes the exact unset commands for PowerShell and bash/zsh.
## Quick start
```bash theme={null}
olostep login
olostep search "best web scraping APIs 2025" --limit 5
olostep answer "What does Olostep do?"
olostep map "https://example.com" --top-n 20
olostep scrape "https://example.com" --formats markdown
olostep crawl "https://docs.example.com" --max-pages 50
olostep batch-scrape urls.csv --formats markdown,html
```
Every command prints its JSON result to stdout by default. Pass `--out ` to save to a file.
## What can it do?
| You want to… | Command | Olostep product |
| ------------------------- | ------------------------------- | ----------------------------------------------- |
| Search the web | `search` | [Searches](/features/search) |
| Get a researched answer | `answer` | [Answers](/features/answers) |
| Discover URLs on a site | `map` | [Maps](/features/maps) |
| Pull one page | `scrape` | [Scrapes](/features/scrapes) |
| Pull every page on a site | `crawl` | [Crawls](/features/crawls) |
| Pull many URLs from a CSV | `batch-scrape` | [Batches](/features/batches) |
| Extract structured fields | `--parser-id` on `batch-scrape` | [Parsers](/features/structured-content/parsers) |
| Refetch a result by ID | `scrape-get` | [Scrapes](/features/scrapes) |
| Tag/organize a batch | `batch-update` | [Batches](/features/batches) |
## Output
Every command **prints its JSON result to stdout** by default.
| Flag | Behavior |
| -------------- | ------------------------------------------ |
| *(none)* | Print JSON to **stdout** (UTF-8, indented) |
| `--out ` | Write JSON to that file instead |
| `--out -` | Explicitly stdout (same as default) |
Progress and log lines go to **stderr**, so stdout stays clean for pipes.
```bash theme={null}
olostep map "https://example.com" --top-n 20 | jq '.urls[:10]'
olostep scrape "https://example.com" | jq .result.markdown_content
olostep search "topic" --json | jq '.links[].url'
```
**Choosing between them:**
* **`search`** — you want a list of relevant URLs and snippets for a query. The CLI searches the web for you.
* **`answer`** — you want a synthesized answer, not raw page content. The CLI does the research for you.
* **`scrape`** — you already have the URL and want clean content out.
* **`crawl`** — you want every page on a site (or a filtered subset) without enumerating URLs by hand.
* **`batch-scrape`** — you have a list of URLs and want them processed in parallel.
## Commands
Use `olostep --help` for every option.
### `search`: live web search
Returns deduplicated organic links (URL, title, description).
| Option | Description |
| ------------------- | ---------------------------------------------- |
| `--limit` | Number of results, default 12, max 25 |
| `--include-domains` | Comma-separated domains to restrict results to |
| `--exclude-domains` | Comma-separated domains to exclude |
| `--out` | File or `-` |
| `--json` | Machine-readable output |
```bash theme={null}
olostep search "TypeScript CLI tools" --limit 10
olostep search "open source projects" --include-domains "github.com" --limit 5
olostep search "AI agents" --json | jq '.links[].url'
```
### `answer`: researched answer
Synchronous — returns when the answer is ready.
| Option | Description |
| --------------- | ----------------------------------------- |
| `--out` | File or `-` |
| `--json-format` | Optional JSON shape for structured output |
```bash theme={null}
olostep answer "What does this company build?" --out answer.json
olostep answer "Extract facts" --json-format '{"company":"","year":""}' --out -
```
### `map`: discover URLs
| Option | Description |
| ------------------------------------------------ | --------------------------------- |
| `--out` | File path or `-` |
| `--top-n` | Max URLs to return |
| `--search-query` | Optional query to guide discovery |
| `--include-subdomain` / `--no-include-subdomain` | Subdomains |
| `--include-url` / `--exclude-url` | Repeatable URL patterns |
| `--cursor` | Pagination cursor |
```bash theme={null}
olostep map "https://example.com" --top-n 100 --search-query "blog"
```
### `scrape`: one URL
**Formats:** `html`, `markdown`, `text`, `json`, `raw_pdf`, `screenshot` (comma-separated; default `markdown`).
| Option | Description |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `--formats` | Comma-separated |
| `--country` | Country code (e.g. `US`, `GB`) |
| `--wait-before-scraping` | Wait before scrape (ms) |
| `--payload-json` / `--payload-file` | Advanced options as JSON (e.g. `"max_age": 86400` to opt into caching — see [Caching](/features/scrapes#caching)) |
```bash theme={null}
olostep scrape "https://example.com" --formats markdown,html
olostep scrape "https://example.com" --payload-file options.json --out -
```
### `scrape-get`: fetch by ID
```bash theme={null}
olostep scrape-get "scrape_abc123" --out -
```
### `crawl`: whole site
Starts a crawl, polls until finished, then retrieves page contents.
**Retrieve formats:** `markdown`, `html`, `json`.
Notable flags: `--max-pages`, `--max-depth`, `--include-subdomain`, `--include-external`, `--include-url`, `--exclude-url`, `--search-query`, `--top-n`, `--webhook`, `--crawl-timeout`, `--formats`, `--pages-limit`, `--pages-search-query`, `--poll-seconds`, `--poll-timeout`, `--dry-run`.
```bash theme={null}
olostep crawl "https://docs.example.com" --max-pages 50 --formats markdown,html
olostep crawl "https://example.com" --max-pages 10 --dry-run
```
### `batch-scrape`: CSV
CSV must have a header row with **`custom_id`** (or `id`) and **`url`** columns.
```csv theme={null}
custom_id,url
example,https://example.com
iana,https://iana.org
docs,https://docs.olostep.com
```
| Option | Description |
| ------------------------------------------------ | -------------------------------------------- |
| `--formats` | `markdown`, `html`, `json` (comma-separated) |
| `--country` | Optional country code |
| `--parser-id` | Parser ID for structured extraction |
| `--poll-seconds`, `--log-every`, `--items-limit` | Polling and paging |
| `--dry-run` | Print payload and exit |
```bash theme={null}
olostep batch-scrape urls.csv --formats markdown,html
olostep batch-scrape urls.csv --parser-id "" --out results.json
```
Synchronous — polls until the batch completes, then retrieves every item.
### `batch-update`: batch metadata
Requires **one of** `--metadata-json` or `--metadata-file` (JSON object).
```bash theme={null}
olostep batch-update "batch_abc123" --metadata-json '{"team":"growth"}'
olostep batch-update "batch_abc123" --metadata-file meta.json
```
## Auth commands
```bash theme={null}
olostep login # browser PKCE sign-in
olostep logout # remove saved credentials
olostep status # show auth state, config paths, version
olostep auth login # same as olostep login
olostep auth logout # same as olostep logout
olostep auth status # same as olostep status
olostep auth set-key # save an API key directly (no browser)
```
`auth set-key` is useful for CI and scripts — write the key directly without going through the browser flow.
## Install the MCP server
The CLI writes the Olostep MCP server into your agent's config — no JSON editing.
```bash theme={null}
olostep mcp install # detect agents, hosted endpoint
olostep mcp install --agent cursor # only Cursor
olostep mcp install --transport stdio # local npx instead of hosted
olostep mcp install --no-global # write into current project
olostep mcp install --dry-run --json # plan only
olostep mcp uninstall # remove the olostep entry
olostep list mcp # show which agents have it
```
| Option | Description |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `--agent` | Specific agent, repeatable. Supported: `cursor`, `claude`, `claude-desktop`, `windsurf`, `vscode`, `kilo`, `opencode`, `continue`, `codex` |
| `--all-agents` / `--no-all-agents` | Target every detected agent (default) |
| `--transport` | `http` (hosted, recommended) or `stdio` (local `npx olostep-mcp`) |
| `--global` / `--no-global` | Per-user config (default) vs project-local |
| `--api-key` | Key to embed; defaults to resolved credentials |
| `--dry-run` | Show the plan without writing |
| `--json` | Machine-readable output |
The hosted endpoint at `https://mcp.olostep.com/mcp` uses `Authorization: Bearer ` — no local Node process required. The CLI merges only the `olostep` key into your existing config. Restart your agent after install.
## Skills for AI agents
The CLI ships **13 Olostep skills** — `SKILL.md` files installed into Claude Code, Cursor, and other agents so they know what Olostep can do and when to use it.
```bash theme={null}
olostep add skills # install all into every detected agent
olostep skills install # same (alias)
olostep skills update # re-install / refresh all skills
olostep skills list # see what's installed and where
olostep skills uninstall # remove all skills
```
Filter what gets installed:
```bash theme={null}
olostep add skills --category usage # core web-data skills only
olostep add skills --skill scrape --skill map
olostep add skills --agent cursor --agent claude
```
See [Skills](/features/skills) for the full list and options.
## Health checks
```bash theme={null}
olostep doctor # run all checks
olostep doctor --skip-network # auth + config only, no HTTP calls
olostep doctor --json # NDJSON — one record per check (good for CI)
olostep doctor --fail-on-warn # exit 1 on warnings too
```
Checks: API key present, API key reachable, MCP endpoint reachable, config file exists for each detected agent.
CI usage:
```bash theme={null}
olostep doctor --json --skip-network | jq 'select(.status == "fail")'
```
## Version & updates
```bash theme={null}
olostep version # CLI version, Node version, channel
olostep version --json # machine-readable: { cli, node, channel }
olostep update # update to latest (npm install -g olostep-cli@latest)
olostep update --check # check whether a newer version is available, don't install
```
## Environment variables
| Variable | Effect |
| --------------------------- | -------------------------------------------------------------- |
| `OLOSTEP_API_KEY` | API key |
| `OLOSTEP_API_TOKEN` | API key (legacy alias) |
| `OLOSTEP_JSON=1` | Force JSON output on every command (same as `--json` globally) |
| `OLOSTEP_NO_UPDATE_CHECK=1` | Silence the "update available" notice |
| `OLOSTEP_CLI_CONFIG_DIR` | Override the credentials directory |
## Windows / PowerShell notes
PowerShell tokenizes `,` and `*` differently from bash — quote arguments:
```powershell theme={null}
olostep scrape "https://example.com" --formats "markdown,html"
olostep map "https://example.com" --include-url "/*"
olostep answer "Extract facts" --json-format '{"company":"","year":""}'
```
Single quotes are safest for JSON values (no `$` interpolation).
## See what's installed
```bash theme={null}
olostep list skills # installed Olostep skills and which agents have them
olostep list mcp # which agents have the Olostep MCP server, and the transport
```
## Global flags
| Flag | Description |
| ----------------- | ----------- |
| `-V`, `--version` | Version |
| `-h`, `--help` | Help |
`--out`, `--timeout`, and `--api-key` are available on every data command.
## Security
Keep API keys out of source control; rotate if leaked. `olostep logout` removes the local credentials file and tells you if any env-var sources still hold a key.
## Related
* [Maps](/features/maps) · [Crawls](/features/crawls) · [Batches](/features/batches) · [Answers](/features/answers) · [Searches](/features/search) · [Skills](/features/skills) · [MCP Server](/integrations/mcp-server)
# Olostep NodeJS SDK
Source: https://docs.olostep.com/sdks/node-js
Official NodeJS SDK to search, extract and structure data from the Web
**NPM Package**: [olostep](https://www.npmjs.com/package/olostep)
## Getting started
```bash theme={null}
npm install olostep
```
```ts camelCase theme={null}
import Olostep from 'olostep';
const client = new Olostep({apiKey: process.env.OLOSTEP_API_KEY});
// Minimal scrape example
const result = await client.scrapes.create('https://example.com');
console.log(result.id, result.html_content);
```
```ts snake_case theme={null}
import Olostep from 'olostep';
const client = new Olostep({api_key: process.env.OLOSTEP_API_KEY});
// Minimal scrape example
const result = await client.scrapes.create('https://example.com');
console.log(result.id, result.html_content);
```
The NodeJS SDK accepts both **camelCase** and **snake\_case** for all parameters. Use **snake\_case** if you're building for AI agents, it matches the API's native field names.
## Usage
### Scraping
Scrape a single URL with various options:
```ts camelCase theme={null}
import Olostep, {Format} from 'olostep';
const client = new Olostep({apiKey: 'your_api_key'});
// Simple scrape
const scrape = await client.scrapes.create('https://example.com');
// With multiple formats
const scrape = await client.scrapes.create({
url: 'https://example.com',
formats: [Format.HTML, Format.MARKDOWN, Format.TEXT],
waitBeforeScraping: 1000,
removeImages: true
});
// Access the content
console.log(scrape.html_content);
console.log(scrape.markdown_content);
// Get scrape by ID
const fetched = await client.scrapes.get(scrape.id);
```
```ts snake_case theme={null}
import Olostep, {Format} from 'olostep';
const client = new Olostep({api_key: 'your_api_key'});
// Simple scrape
const scrape = await client.scrapes.create('https://example.com');
// With multiple formats
const scrape = await client.scrapes.create({
url: 'https://example.com',
formats: [Format.HTML, Format.MARKDOWN, Format.TEXT],
wait_before_scraping: 1000,
remove_images: true
});
// Access the content
console.log(scrape.html_content);
console.log(scrape.markdown_content);
// Get scrape by ID
const fetched = await client.scrapes.get(scrape.id);
```
### Batch Processing
Process multiple URLs in a single batch:
```ts camelCase theme={null}
// Using URL strings (custom IDs auto-generated)
const batch = await client.batches.create([
'https://example.com',
'https://example.org',
'https://example.net'
]);
// Or with explicit custom IDs
const batch = await client.batches.create([
{url: 'https://example.com', customId: 'site-1'},
{url: 'https://example.org', customId: 'site-2'}
]);
console.log(`Batch ${batch.id} created with ${batch.total_urls} URLs`);
// Wait for completion
await batch.waitTillDone({
checkEveryNSecs: 5,
timeoutSeconds: 120
});
// Get batch info
const info = await batch.info();
console.log(info);
// Stream individual results
for await (const item of batch.items()) {
console.log(item.custom_id);
}
```
```ts snake_case theme={null}
// Using URL strings (custom IDs auto-generated)
const batch = await client.batches.create([
'https://example.com',
'https://example.org',
'https://example.net'
]);
// Or with explicit custom IDs
const batch = await client.batches.create([
{url: 'https://example.com', custom_id: 'site-1'},
{url: 'https://example.org', custom_id: 'site-2'}
]);
console.log(`Batch ${batch.id} created with ${batch.total_urls} URLs`);
// Wait for completion
await batch.waitTillDone({
check_every_n_secs: 5,
timeout_seconds: 120
});
// Get batch info
const info = await batch.info();
console.log(info);
// Stream individual results
for await (const item of batch.items()) {
console.log(item.custom_id);
}
```
### Crawling
Crawl an entire website:
```ts camelCase theme={null}
const crawl = await client.crawls.create({
url: 'https://example.com',
maxPages: 100,
maxDepth: 3,
includeUrls: ['*/blog/*'],
excludeUrls: ['*/admin/*']
});
console.log(`Crawl ${crawl.id} started`);
// Wait for completion
await crawl.waitTillDone({
checkEveryNSecs: 10,
timeoutSeconds: 300
});
// Get crawl info
const info = await crawl.info();
console.log(`Crawled ${info.pages_crawled} pages`);
// Stream crawled pages
for await (const page of crawl.pages()) {
console.log(page.url, page.status_code);
}
```
```ts snake_case theme={null}
const crawl = await client.crawls.create({
url: 'https://example.com',
max_pages: 100,
max_depth: 3,
include_urls: ['*/blog/*'],
exclude_urls: ['*/admin/*']
});
console.log(`Crawl ${crawl.id} started`);
// Wait for completion
await crawl.waitTillDone({
check_every_n_secs: 10,
timeout_seconds: 300
});
// Get crawl info
const info = await crawl.info();
console.log(`Crawled ${info.pages_crawled} pages`);
// Stream crawled pages
for await (const page of crawl.pages()) {
console.log(page.url, page.status_code);
}
```
### Site Mapping
Generate a sitemap of URLs from a website:
```ts camelCase theme={null}
const map = await client.maps.create({
url: 'https://example.com',
topN: 100,
includeSubdomain: true,
searchQuery: 'blog posts'
});
console.log(`Map ${map.id} created`);
// Stream URLs
for await (const url of map.urls()) {
console.log(url);
}
// Get map info
const info = await map.info();
```
```ts snake_case theme={null}
const map = await client.maps.create({
url: 'https://example.com',
top_n: 100,
include_subdomain: true,
search_query: 'blog posts'
});
console.log(`Map ${map.id} created`);
// Stream URLs
for await (const url of map.urls()) {
console.log(url);
}
// Get map info
const info = await map.info();
```
### AI-Powered Answers
Get answers from web pages using AI:
```ts camelCase theme={null}
import Olostep from 'olostep';
const client = new Olostep({apiKey: 'your_api_key'});
// Simple task: pass a string directly
const answer = await client.answers.create(
'What is the main topic of https://example.com?'
);
console.log(answer.answer);
console.log(answer.sources);
// With structured JSON output
const structured = await client.answers.create({
task: 'Extract all product names and prices from https://example.com',
jsonFormat: {
products: [{name: '', price: ''}]
}
});
console.log(structured.json_content);
// Retrieve a previously created answer by ID
const fetched = await client.answers.get(answer.id);
console.log(fetched.answer);
```
```ts snake_case theme={null}
import Olostep from 'olostep';
const client = new Olostep({api_key: 'your_api_key'});
// Simple task: pass a string directly
const answer = await client.answers.create(
'What is the main topic of https://example.com?'
);
console.log(answer.answer);
console.log(answer.sources);
// With structured JSON output
const structured = await client.answers.create({
task: 'Extract all product names and prices from https://example.com',
json_format: {
products: [{name: '', price: ''}]
}
});
console.log(structured.json_content);
// Retrieve a previously created answer by ID
const fetched = await client.answers.get(answer.id);
console.log(fetched.answer);
```
### Content Retrieval
Retrieve previously scraped content:
```ts theme={null}
// Get content in specific format(s)
const content = await client.retrieve(retrieveId, Format.MARKDOWN);
console.log(content.markdown_content);
// Multiple formats
const content = await client.retrieve(retrieveId, [
Format.HTML,
Format.MARKDOWN
]);
```
### Advanced Options
#### Custom Actions
Perform browser actions before scraping:
```ts theme={null}
const scrape = await client.scrapes.create({
url: 'https://example.com',
actions: [
{type: 'wait', milliseconds: 2000},
{type: 'click', selector: '#load-more'},
{type: 'scroll', distance: 1000},
{type: 'fill_input', selector: '#search', value: 'query'}
]
});
```
#### Geographic Location
Scrape from different countries using predefined country codes or any valid country code string:
```ts camelCase theme={null}
import Olostep, {Country} from 'olostep';
const client = new Olostep({apiKey: 'your_api_key'});
// Using predefined enum values (US, DE, FR, GB, SG)
const scrape = await client.scrapes.create({
url: 'https://example.com',
country: Country.DE // Germany
});
// Or use any valid country code as a string
const scrape2 = await client.scrapes.create({
url: 'https://example.com',
country: 'jp' // Japan
});
```
```ts snake_case theme={null}
import Olostep, {Country} from 'olostep';
const client = new Olostep({api_key: 'your_api_key'});
// Using predefined enum values (US, DE, FR, GB, SG)
const scrape = await client.scrapes.create({
url: 'https://example.com',
country: Country.DE // Germany
});
// Or use any valid country code as a string
const scrape2 = await client.scrapes.create({
url: 'https://example.com',
country: 'jp' // Japan
});
```
#### Caching
By default, every scrape request fetches the page fresh (`max_age: 0`). Pass `maxAge` to reuse a recent result with the same parameters and improve response time. Value is in **seconds**; the maximum is 7 days (`604800`). See [Caching](/features/scrapes#caching) for details.
```ts camelCase theme={null}
const scrape = await client.scrapes.create({
url: 'https://example.com',
formats: ['markdown'],
maxAge: 86400 // Accept results up to 1 day old
});
```
```ts snake_case theme={null}
const scrape = await client.scrapes.create({
url: 'https://example.com',
formats: ['markdown'],
max_age: 86400 // Accept results up to 1 day old
});
```
#### LLM Extraction
Extract structured data using LLMs:
```ts camelCase theme={null}
const scrape = await client.scrapes.create({
url: 'https://example.com',
llmExtract: {
schema: {
title: 'string',
price: 'number',
description: 'string'
},
prompt: 'Extract product information from this page'
}
});
```
```ts snake_case theme={null}
const scrape = await client.scrapes.create({
url: 'https://example.com',
llm_extract: {
schema: {
title: 'string',
price: 'number',
description: 'string'
},
prompt: 'Extract product information from this page'
}
});
```
### Client Configuration
```ts camelCase theme={null}
import Olostep from 'olostep';
const client = new Olostep({
apiKey: 'your_api_key',
apiBaseUrl: 'https://api.olostep.com/v1', // optional
timeoutMs: 150000, // 150 seconds (optional)
retry: {
maxRetries: 3,
initialDelayMs: 1000
},
userAgent: 'MyApp/1.0' // optional
});
```
```ts snake_case theme={null}
import Olostep from 'olostep';
const client = new Olostep({
api_key: 'your_api_key',
api_base_url: 'https://api.olostep.com/v1', // optional
timeout_ms: 150000, // 150 seconds (optional)
retry: {
max_retries: 3,
initial_delay_ms: 1000
},
user_agent: 'MyApp/1.0' // optional
});
```
### Feature highlights
* Async-first client with full TypeScript support.
* Type-safe inputs using TypeScript enums and interfaces (Formats, Countries, Actions, etc.).
* Rich resource namespaces with both shorthand calls (`client.scrapes.create()`) and explicit methods (`client.scrapes.get()`).
* Shared transport layer with retries, timeouts, and JSON decoding.
* Comprehensive error hierarchy
# Overview
Source: https://docs.olostep.com/sdks/overview
Official Olostep SDKs for Python and Node.js — search, scrape, crawl, and batch the web.
Olostep SDKs are wrappers around the Olostep API to help you easily search, extract and structure data from websites.
## Official SDKs
Explore the official Python SDK for Olostep.
Explore the official NodeJS SDK for Olostep.
Run Olostep from the terminal with JSON output for scripts, CI, and agents.
# Olostep Python SDK
Source: https://docs.olostep.com/sdks/python
Official Python SDK to search, extract and structure data from the Web
**PyPI Package**: [olostep](https://pypi.org/project/olostep/) | **Requirements**: Python 3.11+
## Installation
```bash pip theme={null}
pip install olostep
```
```bash pip3 theme={null}
pip3 install olostep
```
```bash poetry theme={null}
poetry add olostep
```
```bash uv theme={null}
uv pip install olostep
```
## Authentication
Get your API key from the [Olostep Dashboard](https://www.olostep.com/dashboard/).
## Quick Start
The SDK provides two client options depending on your use case:
Best for: Scripts and simple use cases where you prefer blocking operations.
The sync client provides a simpler, blocking interface that's easier to get started with if you're new to async/await.
Best for: Production applications, and handling many concurrent requests.
The async client provides non-blocking operations and is the recommended choice for production applications that need high throughput.
# Sync Client (Olostep)
The sync client (`Olostep`) provides a blocking interface that's perfect for scripts and simple use cases.
```python theme={null}
from olostep import Olostep
# Provide the API key either via passing in the 'api_key' parameter or
# by setting the OLOSTEP_API_KEY environment variable
# The sync client handles resource management automatically
# No explicit close needed - resources are cleaned up after each operation
client = Olostep(api_key="YOUR_REAL_KEY")
scrape_result = client.scrapes.create(url_to_scrape="https://example.com")
```
### Basic Web Scraping
```python theme={null}
from olostep import Olostep
client = Olostep(api_key="your-api-key")
# Simple scraping
result = client.scrapes.create(url_to_scrape="https://example.com")
print(f"Scraped {len(result.html_content)} characters")
# Multiple formats
result = client.scrapes.create(
url_to_scrape="https://example.com",
formats=["html", "markdown"]
)
print(f"HTML: {len(result.html_content)} chars")
print(f"Markdown: {len(result.markdown_content)} chars")
```
### Batch Processing
```python theme={null}
from olostep import Olostep
client = Olostep(api_key="your-api-key")
# Process multiple URLs efficiently
batch = client.batches.create(
urls=[
"https://www.google.com/search?q=python",
"https://www.google.com/search?q=javascript",
"https://www.google.com/search?q=typescript"
]
)
# Wait for completion and process results
for item in batch.items():
content = item.retrieve(["html"])
print(f"Processed {item.url}: {len(content.html_content)} bytes")
```
### Smart Web Crawling
```python theme={null}
from olostep import Olostep
client = Olostep(api_key="your-api-key")
# Crawl with intelligent filtering
crawl = client.crawls.create(
start_url="https://www.bbc.com",
max_pages=100,
include_urls=["/articles/**", "/blog/**"],
exclude_urls=["/admin/**"]
)
for page in crawl.pages():
content = page.retrieve(["html"])
print(f"Crawled: {page.url}")
```
### Site Mapping
```python theme={null}
from olostep import Olostep
client = Olostep(api_key="your-api-key")
# Extract all links from a website
maps = client.maps.create(url="https://example.com")
# Get all discovered URLs
urls = []
for url in maps.urls():
urls.append(url)
if len(urls) >= 10: # Limit for demo
break
print(f"Found {len(urls)} URLs")
```
### AI-Powered Answers
```python theme={null}
from olostep import Olostep
client = Olostep(api_key="your-api-key")
# Get answers from web pages using AI
answer = client.answers.create(
task="What is the main topic of https://example.com?"
)
print(f"Answer: {answer.answer}")
```
# Async Client (AsyncOlostep)
The async client (`AsyncOlostep`) is the recommended client for high-performance applications, backend services, and when you need to handle many concurrent requests.
```python theme={null}
from olostep import AsyncOlostep
# Provide the API key either via passing in the 'api_key' parameter or
# by setting the OLOSTEP_API_KEY environment variable
# RESOURCE MANAGEMENT
# ===================
# The SDK supports two usage patterns for resource management:
# 1. Context Manager (Recommended for one-off usage):
# Automatically handles resource cleanup
async with AsyncOlostep(api_key="YOUR_REAL_KEY") as client:
scrape_result = await client.scrapes.create(url_to_scrape="https://example.com")
# Transport is automatically closed here
# 2. Explicit Close (For long-lived services):
# Requires manual resource cleanup
client = AsyncOlostep(api_key="YOUR_REAL_KEY")
try:
scrape_result = await client.scrapes.create(url_to_scrape="https://example.com")
finally:
await client.close() # Manually close the transport
```
### Basic Web Scraping
```python theme={null}
import asyncio
from olostep import AsyncOlostep
async def main():
async with AsyncOlostep(api_key="your-api-key") as client:
# Simple scraping
result = await client.scrapes.create(url_to_scrape="https://example.com")
print(f"Scraped {len(result.html_content)} characters")
# Multiple formats
result = await client.scrapes.create(
url_to_scrape="https://example.com",
formats=["html", "markdown"]
)
print(f"HTML: {len(result.html_content)} chars")
print(f"Markdown: {len(result.markdown_content)} chars")
asyncio.run(main())
```
### Batch Processing
```python theme={null}
import asyncio
from olostep import AsyncOlostep
async def main():
async with AsyncOlostep(api_key="your-api-key") as client:
# Process multiple URLs efficiently
batch = await client.batches.create(
urls=[
"https://www.google.com/search?q=python",
"https://www.google.com/search?q=javascript",
"https://www.google.com/search?q=typescript"
]
)
# Wait for completion and process results
async for item in batch.items():
content = await item.retrieve(["html"])
print(f"Processed {item.url}: {len(content.html_content)} bytes")
asyncio.run(main())
```
### Smart Web Crawling
```python theme={null}
import asyncio
from olostep import AsyncOlostep
async def main():
async with AsyncOlostep(api_key="your-api-key") as client:
# Crawl with intelligent filtering
crawl = await client.crawls.create(
start_url="https://www.bbc.com",
max_pages=100,
include_urls=["/articles/**", "/blog/**"],
exclude_urls=["/admin/**"]
)
async for page in crawl.pages():
content = await page.retrieve(["html"])
print(f"Crawled: {page.url}")
asyncio.run(main())
```
### Site Mapping
```python theme={null}
import asyncio
from olostep import AsyncOlostep
async def main():
async with AsyncOlostep(api_key="your-api-key") as client:
# Extract all links from a website
maps = await client.maps.create(url="https://example.com")
# Get all discovered URLs
urls = []
async for url in maps.urls():
urls.append(url)
if len(urls) >= 10: # Limit for demo
break
print(f"Found {len(urls)} URLs")
asyncio.run(main())
```
### AI-Powered Answers
```python theme={null}
import asyncio
from olostep import AsyncOlostep
async def main():
async with AsyncOlostep(api_key="your-api-key") as client:
# Get answers from web pages using AI
answer = await client.answers.create(
task="What is the main topic of https://example.com?"
)
print(f"Answer: {answer.answer}")
asyncio.run(main())
```
## SDK Reference
### Method Structure
Both SDK clients provide the same clean, pythonic interface organized into logical namespaces:
| Namespace | Purpose | Key Methods |
| ---------- | --------------------- | ------------------------------- |
| `scrapes` | Single URL extraction | `create()`, `get()` |
| `batches` | Multi-URL processing | `create()`, `info()`, `items()` |
| `crawls` | Website traversal | `create()`, `info()`, `pages()` |
| `maps` | Link extraction | `create()`, `urls()` |
| `answers` | AI-powered extraction | `create()`, `get()` |
| `retrieve` | Content retrieval | `get()` |
Each operation returns stateful objects with ergonomic methods for follow-up operations.
## Error Handling
Catch all SDK errors using the base exception class:
```python theme={null}
from olostep import Olostep, Olostep_BaseError
client = Olostep(api_key="your-api-key")
try:
result = client.scrapes.create(url_to_scrape="https://example.com")
except Olostep_BaseError as e:
print(f"Error has occurred: {type(e).__name__}")
print(f"Error message: {e}")
```
For detailed error handling information, including the full exception hierarchy and granular error handling options, see [Detailed Error Handling](/sdks/python#detailed-error-handling).
## Automatic Retries
The SDK automatically retries on transient errors (network issues, temporary server problems) based on the `RetryStrategy` configuration. You can customize the retry behavior by passing a `RetryStrategy` instance when creating the client:
```python theme={null}
from olostep import Olostep, RetryStrategy
retry_strategy = RetryStrategy(
max_retries=3,
initial_delay=1.0,
jitter_min=0.2,
jitter_max=0.8
)
client = Olostep(api_key="your-api-key", retry_strategy=retry_strategy)
result = client.scrapes.create("https://example.com")
```
For detailed retry configuration options and best practices, see [Retry Strategy](/sdks/python#retry-strategy-configuration).
## Advanced Features
### Smart Input Coercion
The SDK intelligently handles various input formats for maximum convenience:
```python theme={null}
from olostep import Olostep, Country
client = Olostep(api_key="your-api-key")
# Formats: string, list, or enum
client.scrapes.create(url_to_scrape="https://example.com", formats="html")
client.scrapes.create(url_to_scrape="https://example.com", formats=["html", "markdown"])
# Countries: case-insensitive strings or enums
client.scrapes.create(url_to_scrape="https://example.com", country="us")
client.scrapes.create(url_to_scrape="https://example.com", country=Country.US)
# Lists: single values or lists
client.batches.create(urls="https://example.com") # Single URL
client.batches.create(urls=["https://a.com", "https://b.com"]) # Multiple URLs
```
### Advanced Scraping Options
```python theme={null}
from olostep import Olostep, Format, Country, WaitAction, FillInputAction
client = Olostep(api_key="your-api-key")
# Full control over scraping behavior
result = client.scrapes.create(
url_to_scrape="https://news.google.com/",
wait_before_scraping=3000,
formats=[Format.HTML, Format.MARKDOWN],
remove_css_selectors=["script", ".popup"],
actions=[
WaitAction(milliseconds=1500),
FillInputAction(selector="searchbox", value="olostep")
],
parser="@olostep/google-news",
country=Country.US,
remove_images=True
)
```
### Caching
By default, every scrape request fetches the page fresh (`max_age=0`). Pass `max_age` to reuse a recent result with the same parameters and improve response time. Value is in **seconds**; the maximum is 7 days (`604800`). See [Caching](/features/scrapes#caching) for details.
```python theme={null}
# Opt-in to caching: accept results up to 1 day (86400 seconds) old
result = client.scrapes.create(
url_to_scrape="https://example.com",
formats=["markdown"],
max_age=86400
)
```
### Batch Processing with Custom IDs
```python theme={null}
from olostep import Olostep, Country
client = Olostep(api_key="your-api-key")
batch = client.batches.create([
{"url": "https://www.google.com/search?q=python", "custom_id": "search_1"},
{"url": "https://www.google.com/search?q=javascript", "custom_id": "search_2"},
{"url": "https://www.google.com/search?q=typescript", "custom_id": "search_3"}
],
country=Country.US,
parser="@olostep/google-search"
)
# Process results by custom ID
# When using a parser, retrieve JSON content instead of HTML
for item in batch.items():
if item.custom_id == "search_2":
content = item.retrieve(["json"])
print(f"Search result: {content.json_content}")
```
### Intelligent Crawling
```python theme={null}
from olostep import Olostep
client = Olostep(api_key="your-api-key")
# Crawl with intelligent filtering
crawl = client.crawls.create(
start_url="https://www.bbc.com",
max_pages=1000,
max_depth=3,
include_urls=["/articles/**", "/news/**"],
exclude_urls=["/ads/**", "/tracking/**"],
include_external=False,
include_subdomain=True,
)
for page in crawl.pages():
content = page.retrieve(["html"])
print(f"Crawled: {page.url}")
```
### Site Mapping with Filters
```python theme={null}
from olostep import Olostep
client = Olostep(api_key="your-api-key")
# Extract all links with advanced filtering
maps = client.maps.create(
url="https://www.bbc.com",
include_subdomain=True,
include_urls=["/articles/**", "/news/**"],
exclude_urls=["/ads/**", "/tracking/**"]
)
# Get filtered URLs
urls = []
for url in maps.urls():
urls.append(url)
print(f"Found {len(urls)} relevant URLs")
```
### Answers Retrieval
```python theme={null}
from olostep import Olostep
client = Olostep(api_key="your-api-key")
# First create an answer
created_answer = client.answers.create(
task="What is the main topic of https://example.com?"
)
# Then retrieve it using the ID
answer = client.answers.get(answer_id=created_answer.id)
print(f"Answer: {answer.answer}")
```
### Content Retrieval
```python theme={null}
from olostep import Olostep
client = Olostep(api_key="your-api-key")
# Get content by retrieve ID
result = client.retrieve.get(retrieve_id="ret_123")
# Get multiple formats
result = client.retrieve.get(retrieve_id="ret_123", formats=["html", "markdown", "text", "json"])
```
## Logging
Enable logging to debug issues:
```python theme={null}
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("olostep")
logger.setLevel(logging.INFO) # Use DEBUG for verbose output
```
**Log Levels**: `INFO` (recommended), `DEBUG` (verbose), `WARNING`, `ERROR`
## Retry Strategy Configuration
The `RetryStrategy` class controls how the Olostep SDK handles transient API errors through automatic retries with exponential backoff and jitter. This helps ensure reliable operation in production environments where temporary network issues, rate limits, and server overload can cause intermittent failures.
### Default Behavior
By default, the SDK uses the following retry configuration:
* **Max retries**: 5 attempts
* **Initial delay**: 2 seconds
* **Backoff**: Exponential (2^attempt)
* **Jitter**: 10-90% of delay (randomized)
This means:
* Attempt 1: Immediate
* Attempt 2: \~2-3.6s delay
* Attempt 3: \~4-7.2s delay
* Attempt 4: \~8-14.4s delay
* Attempt 5: \~16-28.8s delay
Maximum duration: \~57 seconds for all retries (worst case)
### Custom Configuration
```python theme={null}
from olostep import AsyncOlostep, RetryStrategy
# Create custom retry strategy
retry_strategy = RetryStrategy(
max_retries=3,
initial_delay=1.0,
jitter_min=0.2, # 20% minimum jitter
jitter_max=0.8, # 80% maximum jitter
)
# Use with client
async with AsyncOlostep(
api_key="your-api-key",
retry_strategy=retry_strategy
) as client:
result = await client.scrapes.create("https://example.com")
```
### When Retries Happen
The SDK automatically retries on:
* **Temporary server issues** (`OlostepServerError_TemporaryIssue`)
* **Timeout responses** (`OlostepServerError_NoResultInResponse`)
Other errors (authentication, validation, resource not found, etc.) fail immediately without retry.
### Transport vs Caller Retries
The SDK has two retry layers:
1. **Transport layer**: Handles network-level connection failures (DNS, timeouts, etc.)
2. **Caller layer**: Handles API-level transient errors (controlled by `RetryStrategy`)
Both layers are independent and have separate configuration. The total maximum duration is the sum of both layers.
### Calculating Max Duration
```python theme={null}
retry_strategy = RetryStrategy(max_retries=5, initial_delay=2.0)
max_duration = retry_strategy.max_duration()
print(f"Max call duration: {max_duration:.2f}s")
```
### Configuration Examples
Here are some examples of how to configure the retry strategy for different use cases.
#### Conservative Strategy
```python theme={null}
# Fewer retries, shorter delays
retry_strategy = RetryStrategy(
max_retries=3,
initial_delay=1.0,
jitter_min=0.2,
jitter_max=0.8
)
# Max duration: ~12.6s
```
#### Aggressive Strategy
```python theme={null}
# More retries for critical operations
retry_strategy = RetryStrategy(
max_retries=10,
initial_delay=0.5
)
# Max duration: ~969.75s
```
#### No Retries (Fail Fast)
```python theme={null}
# Disable retries for immediate failure feedback
retry_strategy = RetryStrategy(max_retries=0)
client = AsyncOlostep(api_key="your-api-key", retry_strategy=retry_strategy)
```
#### High-Throughput Strategy
```python theme={null}
# Optimized for high-volume operations
retry_strategy = RetryStrategy(
max_retries=2,
initial_delay=0.5,
jitter_min=0.1,
jitter_max=0.3 # Lower jitter for more predictable timing
)
# Max duration: ~1.95s
```
### Understanding Jitter
Jitter adds randomization to prevent "thundering herd" problems when many clients retry simultaneously. The jitter is calculated as:
```python theme={null}
base_delay = initial_delay * (2 ** attempt)
jitter_range = base_delay * (jitter_max - jitter_min)
jitter = random.uniform(base_delay * jitter_min, base_delay * jitter_min + jitter_range)
final_delay = base_delay + jitter
```
For example, with `initial_delay=2.0`, `jitter_min=0.1`, `jitter_max=0.9`:
* Attempt 0: base=2.0s, jitter=0.2-1.8s, final=2.2-3.8s
* Attempt 1: base=4.0s, jitter=0.4-3.6s, final=4.4-7.6s
* Attempt 2: base=8.0s, jitter=0.8-7.2s, final=8.8-15.2s
### Best Practices
#### For Production Applications
```python theme={null}
# Balanced approach for production
retry_strategy = RetryStrategy(
max_retries=5,
initial_delay=2.0,
jitter_min=0.1,
jitter_max=0.9
)
```
#### For Development/Testing
```python theme={null}
# Fast feedback for development
retry_strategy = RetryStrategy(
max_retries=2,
initial_delay=0.5,
jitter_min=0.1,
jitter_max=0.3
)
```
#### For Batch Operations
```python theme={null}
# Conservative for large batch jobs
retry_strategy = RetryStrategy(
max_retries=3,
initial_delay=1.0,
jitter_min=0.2,
jitter_max=0.8
)
```
### Monitoring and Debugging
The SDK logs retry information at the DEBUG level:
```
DEBUG: Temporary issue, retrying in 2.34s
DEBUG: No result in response, retrying in 4.67s
```
Enable debug logging to monitor retry behavior:
```python theme={null}
import logging
logging.getLogger("olostep").setLevel(logging.DEBUG)
```
### Error Handling
When all retries are exhausted, the original error is raised:
```python theme={null}
try:
result = await client.scrapes.create("https://example.com")
except OlostepServerError_TemporaryIssue as e:
print(f"Failed after all retries: {e}")
# Handle the permanent failure
```
### Performance Considerations
* **Memory**: Each retry attempt uses additional memory for request/response objects
* **Time**: Total operation time can be significantly longer with retries enabled
* **API Limits**: Retries count against your API usage limits
* **Network**: More network traffic due to retry attempts
Choose your retry strategy based on your application's requirements for reliability vs. performance.
## Detailed Error Handling
### Exception Hierarchy
The Olostep SDK provides a comprehensive exception hierarchy for different failure scenarios. All exceptions inherit from `Olostep_BaseError`.
There are three main error types that directly inherit from `Olostep_BaseError`:
1. **`Olostep_APIConnectionError`** - Network-level connection failures
2. **`OlostepServerError_BaseError`** - Errors raised (sort of) by the API server
3. **`OlostepClientError_BaseError`** - Errors raised by the client SDK
### Why Connection Errors Are Separate
`Olostep_APIConnectionError` is separate from server errors because it represents network-level failures that occur before the API can process the request. These are transport layer issues (DNS or HTTP failures, timeouts, connection refused, etc.) rather than API-level errors. HTTP status codes (4xx, 5xx) are considered API responses and are categorized as server errors, even though they indicate problems.
```
Olostep_BaseError
├── Olostep_APIConnectionError
├── OlostepServerError_BaseError
│ ├── OlostepServerError_TemporaryIssue
│ │ ├── OlostepServerError_NetworkBusy
│ │ └── OlostepServerError_InternalNetworkIssue
│ ├── OlostepServerError_RequestUnprocessable
│ │ ├── OlostepServerError_ParserNotFound
│ │ └── OlostepServerError_OutOfResources
│ ├── OlostepServerError_BlacklistedDomain
│ ├── OlostepServerError_FeatureApprovalRequired
│ ├── OlostepServerError_AuthFailed
│ ├── OlostepServerError_CreditsExhausted
│ ├── OlostepServerError_InvalidEndpointCalled
│ ├── OlostepServerError_ResourceNotFound
│ ├── OlostepServerError_NoResultInResponse
│ └── OlostepServerError_UnknownIssue
└── OlostepClientError_BaseError
├── OlostepClientError_RequestValidationFailed
├── OlostepClientError_ResponseValidationFailed
├── OlostepClientError_NoAPIKey
├── OlostepClientError_AsyncContext
├── OlostepClientError_BetaFeatureAccessRequired
└── OlostepClientError_Timeout
```
### Recommended Error Handling
For most use cases, catch the base error and print the error name:
```python theme={null}
from olostep import AsyncOlostep, Olostep_BaseError
try:
result = await client.scrapes.create(url_to_scrape="https://example.com")
except Olostep_BaseError as e:
print(f"Error has occurred: {type(e).__name__}")
print(f"Error message: {e}")
```
This approach catches all SDK errors and provides clear information about what went wrong. The error name (e.g., `OlostepServerError_AuthFailed`) is descriptive enough to understand the issue.
### Granular Error Handling
If you need more specific error handling, catch the specific error types directly. **Avoid using `OlostepServerError_BaseError` or `OlostepClientError_BaseError`** - these base classes only indicate who raised the error (server vs client), not who's responsible for fixing it. This is an implementation detail that doesn't help with error handling logic.
Instead, catch specific error types that indicate the actual problem:
```python theme={null}
from olostep import (
AsyncOlostep,
Olostep_BaseError,
Olostep_APIConnectionError,
OlostepServerError_AuthFailed,
OlostepServerError_CreditsExhausted,
OlostepClientError_NoAPIKey,
)
try:
result = await client.scrapes.create(url_to_scrape="https://example.com")
except Olostep_APIConnectionError as e:
print(f"Network error: {type(e).__name__}")
except OlostepServerError_AuthFailed:
print("Invalid API key")
except OlostepServerError_CreditsExhausted:
print("Credits exhausted")
except OlostepClientError_NoAPIKey:
print("API key not provided")
except Olostep_BaseError as e:
print(f"Error has occurred: {type(e).__name__}")
```
## Configuration
### Environment Variables
| Variable | Description | Default |
| ---------------------- | ------------------------- | ---------------------------- |
| `OLOSTEP_API_KEY` | Your API key | Required |
| `OLOSTEP_BASE_API_URL` | API base URL | `https://api.olostep.com/v1` |
| `OLOSTEP_API_TIMEOUT` | Request timeout (seconds) | `150` |
## Getting Help
* [Full Documentation](https://docs.olostep.com)
* [Community Slack](https://join.slack.com/t/olostep-users/shared_invite/zt-2pn2ce0uu-~591qIdhAfJy~LXCWQS5UQ)
* [Support Email](mailto:info@olostep.com)
## Resources
View on PyPI
Sign up for free
# Pay-per-use (x402)
Source: https://docs.olostep.com/x402
Pay-per-use API endpoints with stablecoins
## Overview
You can now also use the Olostep API endpoints with pay-per-use payments using stablecoins.
These endpoints use the x402 payment protocol, enabling pay-per-use access with stablecoin payments. Each request requires a payment header for authentication and payment.
**Pay-Per-Use Endpoints**
All endpoints listed below accept stablecoin payments via the x402 protocol. No subscription required - pay only for what you use.
## Authentication
Include the payment header with your requests:
```
X-Payment: {{paymentHeader}}
```
The payment is automatically processed and verified before your request is fulfilled.
***
## Olostep API
### Payment-Enabled Endpoints
#### POST /v1/maps
This endpoint allows users to get all the urls on a certain website. It can take up to 120 seconds for complex websites. For large websites, results are paginated using cursor-based pagination
**Price:** \$0.01 per request
**Network:** base (USDC)
```bash theme={null}
curl -X POST 'https://api.olostep.com/x402/v1/maps' \\
-H 'Content-Type: application/json' \\
-H 'X-Payment: {{paymentHeader}}' \\
-d '{
"url": "example",
"search_query": "example",
"top_n": 123,
"include_subdomain": true,
"include_urls": "",
"exclude_urls": "",
"cursor": "example"
}'
```
#### POST /v1/scrapes
Initiate a web page scrape
**Price:** \$0.01 per request
**Network:** base (USDC)
```bash theme={null}
curl -X POST 'https://api.olostep.com/x402/v1/scrapes' \\
-H 'Content-Type: application/json' \\
-H 'X-Payment: {{paymentHeader}}' \\
-d '{
"url_to_scrape": "example",
"wait_before_scraping": "",
"formats": "",
"remove_css_selectors": "example",
"actions": "",
"country": "example",
"transformer": "example",
"remove_images": true,
"remove_class_names": "",
"parser": "",
"llm_extract": "",
"links_on_page": "",
"screen_size": "",
"metadata": ""
}'
```
#### POST /v1/crawls
Starts a new crawl. You receive a `id` to track the progress. The operation may take 1-10 mins depending upon the site and depth and pages parameters.
**Price:** Dynamic - calculated per request based on usage
This endpoint uses dynamic pricing. The actual cost is determined by your request parameters and will be shown in the 402 Payment Required response before processing.
**Network:** base (USDC)
```bash theme={null}
curl -X POST 'https://api.olostep.com/x402/v1/crawls' \\
-H 'Content-Type: application/json' \\
-H 'X-Payment: {{paymentHeader}}' \\
-d '{
"start_url": "example",
"max_pages": 123,
"include_urls": "",
"exclude_urls": "",
"max_depth": 123,
"include_external": true,
"include_subdomain": true,
"search_query": "example",
"top_n": 123,
"webhook_url": "example",
"timeout": 123
}'
```
#### POST /v1/answers
The AI will perform actions like searching and browsing web pages to find the answer to the provided task. Execution time is 3-30s depending upon complexity. For longer tasks, use the agent endpoint instead.
**Price:** \$0.05 per request
**Network:** base (USDC)
```bash theme={null}
curl -X POST 'https://api.olostep.com/x402/v1/answers' \\
-H 'Content-Type: application/json' \\
-H 'X-Payment: {{paymentHeader}}' \\
-d '{
"task": "example",
"json_format": ""
}'
```
### Standard Endpoints
These endpoints do not require payment:
* **GET** `/v1/crawls/{crawl_id}` - Fetches information about a specific crawl.
* **GET** `/v1/batches/{batch_id}/items` - Retrieves the list of items processed for a batch. You can then use the `retrieve_id` to get the content with the Retrieve Endpoint
* **GET** `/v1/crawls/{crawl_id}/pages` - Fetches the list of pages for a specific crawl.
* **GET** `/v1/batches/{batch_id}` - Retrieves the status and progress information about a batch. To retrieve the content for a batch, see here
* **GET** `/v1/answers/{answer_id}` - This endpoint retrieves a previously completed answer by its ID.
* **GET** `/v1/scrapes/{scrape_id}` - Can be used to retrieve response for a scrape.
* **GET** `/v1/retrieve` - Retrieve page content of processed batches and crawls urls.
***
## How x402 Works
The payment flow is handled automatically by the x402 SDK:
1. **Make Request** - Send a request to the endpoint
2. **Payment Required** - Server responds with payment requirements (402 status)
3. **Auto-Payment** - SDK automatically creates and submits payment
4. **Get Response** - Receive your API response
## Getting Started
Install the x402 SDK for your language:
```bash theme={null}
# Node.js
npm install x402-fetch viem
# Python
pip install x402 eth-account
```
## Example Usage
```javascript Node.js theme={null}
import { wrapFetchWithPayment } from "x402-fetch";
import { privateKeyToAccount } from "viem/accounts";
const account = privateKeyToAccount(process.env.PRIVATE_KEY);
const fetchWithPayment = wrapFetchWithPayment(fetch, account);
// Make a paid request - payment is automatic
const response = await fetchWithPayment("https://api.olostep.com/x402/v1/maps", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ /* your data */ })
});
const result = await response.json();
console.log(result);
```
```python Python theme={null}
import requests
from eth_account import Account
from x402.clients.requests import x402_http_adapter
account = Account.from_key(os.getenv("PRIVATE_KEY"))
session = requests.Session()
adapter = x402_http_adapter(account)
session.mount("https://", adapter)
# Make a paid request - payment is automatic
response = session.post(
"https://api.olostep.com/x402/v1/maps",
json={"key": "value"}
)
print(response.json())
```
## Learn More
* [x402 Protocol Docs](https://x402.org)
* [Coinbase x402 Guide](https://docs.cdp.coinbase.com/x402/)
***
Powered by [Orthogonal](https://orthogonal.sh)