Skip to main content
The Olostep plugin brings web data to Dify agents, chatflows and workflows. Olostep is web data infrastructure for AI agents and enterprise AI workflows. Its APIs provide live web search, scraping and crawling to extract, clean and structure data from the web. Install from the Dify Marketplace →

​
Why Olostep

  • Web data infrastructure for developers and enterprises
  • Scales to hundreds of millions to billions of requests per customer per month
  • The most performant, reliable and cost-effective API on the market
  • Used by the fastest-growing AI companies
From extracting data from a handful of URLs to crawling hundreds of millions of pages, Olostep gives developers and AI companies reliable, scalable and highly cost-effective access to clean, structured web data.

​
Tools

​
Setup

  1. Sign up at olostep.com and copy your API key from the Olostep dashboard.
  2. In Dify, go to Plugins, find Olostep in the Marketplace and click Install.
  3. Open the plugin, click Authorize and paste your API key.
Dify stores the key as a secret. The plugin sends it only to the Olostep API.

​
Usage

​
Agents

Add the Olostep tools to an agent app. The agent picks the tool it needs, for example Search Web to find pages and Scrape URL to read them.

​
Workflows and chatflows

Add a tool node, choose an Olostep tool and map its inputs. Every tool returns JSON for later nodes and readable text for LLM nodes. For example, a research workflow can run Search Web for a topic, then Batch Scrape on the result URLs, then summarize the pages with an LLM node.

​
Crawls and batches

Crawl Website and Batch Scrape wait for results for up to 2 minutes by default. You can raise this to 4 minutes with Wait for results (seconds). If the job is still running when the wait ends, the tool returns its id with status set to in_progress. Pass that id to Get Results later to get the pages. Each call returns up to 100 pages. If more are available, the result includes next_cursor: pass it to Get Results with the id to get the next pages.
Batch Scrape takes up to 100k URLs. New accounts are limited to 100 URLs per batch: contact info@olostep.com to lift this.

​
Self-hosted Dify

The plugin connects only to https://api.olostep.com over HTTPS, so your Dify instance needs outbound network access to that host.

​
Resources