Overview
Olostep provides a web scraping API that enables real-time price tracking of millions of products on an e-commerce at regular intervals (e.g. every few hours) in a scalable and cost effective way.
This is useful for businesses that want to monitor price fluctuations, compare prices across multiple websites, or track competitor pricing strategies.
In this guide, we will see how a customer is using Olostep to set up automated price tracking for millions of Amazon products daily.
Why Use Olostep for Price Tracking?
- Scalability: Track prices for millions of products every few hours.
- Automation: Set up scheduled scraping tasks that run at predefined times/regular intervals.
- Multiple Formats: Retrieve data in JSON, html or markdown format.
- Custom Parsers: Extract only the relevant JSON information with our parsers or pass your own to the API.
How to Track Prices Using Olostep
Overview of the Process Setup
When tracking products at scale we recommend using Olostep’s Batches endpoint.
This endpoint allows you to send multiple batches of URLs (each of up to 10k) to be processed in parallel and then retrieve the results after 5-8 minutes. You can send multiple batches at the same time, monitor their progress and retrieve the results once they are complete. In this way you can process millions of URLs in 15-20 minutes.
The overall flow for price tracking using Olostep is as follows:
- Read the products from the database and save the URLs you want to track in a CSV file.
- Read the data from the CSV file and start a batch using Olostep’s batch endpoint. This is done by posting the data to the endpoint in chunks of up to 10,000 URLs at a time.
- Check the batch status every 60 seconds to monitor the progress.
- Once the batch is complete, read the content and use it in your workflow.
You can start a batch and be returned the html/markdown content of the page and then parse it yourself to extract the data you want. But we recommend starting the batch with a parser so you are returned a JSON object with only the parsed data you need.
You can pass your own parser to the API or use one of the pre-built ones we have for some common websites (e.g Amazon product pages, Google search results, Linkedin profiles, etc…).
We store the data for each batch for 7 days so you can retrieve it multiple times if needed.
Step 1: Export Product Data from your Database
The first step is to retrieve product information from your database and save it in a CSV format. This file should contain product identifiers, URLs, and any additional metadata required for tracking.
Step 2: Start a Batch with Olostep
To start a batch, read the product data from the CSV and send it to the Olostep batch endpoint. This is done using an HTTP POST request with a JSON payload.
Each batch can have up to 10k URLs. For large datasets (>10,000 URLs), split into multiple batches and send them in parallel.
A batch consists of an array of items, where each item represents a product URL to be processed. Here’s the structure of a batch request
import requests
def start_batch(batch_array):
payload = {
"batch_array": batch_array, # Array of items to process
"batch_country": "IT", # Country code for the batch
"parser": "@olostep/amazon-it-product" # Optional: Specify a custom parser so you only get the JSON data you need
}
headers = {
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json"
}
response = requests.post(
"https://api.olostep.com/v1/batches",
headers=headers,
json=payload
)
return response.json()["id"]
Batch Array Structure
Each item in the batch_array should follow this structure:
{
"custom_id": "unique_identifier", // Required: Your unique identifier for the item
"url": "product_url", // Required: URL to be processed
"wait_before_scraping": 0, // Optional: Wait time before scraping each url in the batch (in milliseconds)
}
Array of items to process. Maximum of 10,000 URLs per batch. Each item must have a unique custom_id.
Two-letter country code (e.g., “IT” for Italy).
Name of the custom parser to use (e.g., “@olostep/amazon-it-product”). Contact us at info@olostep.com to get access to the pre-built parsers or to create your own. {
"id": "batch_54ikwskmt8"
}
# Prepare batch array
batch_array = [
{
"custom_id": "product_123",
"url": "https://www.amazon.it/dp/B0CHF6Z393/?coliid=INQXTGFQF4FM4&colid=1R0NGA5NR5LSZ&psc=1&ref_=list_c_wl_lv_vv_lig_dp_it"
},
{
"custom_id": "product_124",
"url": "https://www.amazon.it/dp/B0CHMJL774/?coliid=I6CFYA5EHVHE2&colid=1R0NGA5NR5LSZ&psc=1&ref_=list_c_wl_lv_vv_lig_dp_it"
}
]
# Start batch processing
batch_id = start_batch(batch_array)
print(f"Started batch: {batch_id}")
Step 3: Monitor Batch Status
Once a batch is started, you’ll need to monitor its status to determine when processing is complete. The API provides a status endpoint that can be polled periodically (e.g., every 60 seconds) with the batch_id
import requests
def check_batch_status(batch_id):
headers = {"Authorization": "Bearer " + API_KEY}
response = requests.request(
"GET",
f"https://api.olostep.com/v1/batches/{batch_id}",
headers=headers
)
return response.json()["status"]
import asyncio
async def monitor_batch(batch_id: str) -> None:
"""Monitor a single batch until it's completed"""
while True:
status = check_batch_status(batch_id)
if status == "completed":
print(f"Batch {batch_id} completed!")
return
print(f"Batch {batch_id} still processing... Checking again in 60 seconds")
await asyncio.sleep(60)
Step 4: Retrieve the IDs for Completed Items
Once the batch is marked as completed, you can fetch the list of completed items. Each item will have a retrieve_id. If you want the actual content use the retrieve endpoint by passing the retrieve_id
import requests
def get_completed_items(batch_id):
headers = {"Authorization": "Bearer YOUR_API_KEY"}
response = requests.get(f"https://api.olostep.com/v1/batches/{batch_id}/items", headers=headers)
return response.json()["items"]
retrieve_id for every URL sent. You can then use the retrieve endpoint to retrieve and store the extracted data (html, markdown or JSON) for each URL.
You can get the retrieve_id for each item in the batch using the following code:
items = get_completed_items("your_batch_id")
for item in items:
print(f"""
URL: {item['url']}
Custom ID: {item['custom_id']}
Retrieve ID: {item['retrieve_id']}
---
""")
Step 5: Retrieve the Content for each Item
Once you have the retrieve_id for each item, you can fetch its content (HTML, Markdown, or JSON) using the retrieve endpoint:
def retrieve_content(retrieve_id):
url = "https://api.olostep.com/v1/retrieve"
headers = {"Authorization": "Bearer YOUR_API_KEY"}
params = {"retrieve_id": retrieve_id}
response = requests.get(
url,
headers=headers,
params=params
)
return response.json()
# Example usage
retrieve_id = "product_123"
content = retrieve_content(retrieve_id)
# If you want to process multiple items
def process_batch_content(batch_id):
items = get_completed_items(batch_id)
for item in items:
content = retrieve_content(item['retrieve_id'])
# Process or store the content as needed
Conclusion
By following these steps, you can set up an automated price-tracking system using Olostep. Shortly we will publish an open-source github repo with the full code for this example.
