> ## Documentation Index
> Fetch the complete documentation index at: https://docs.olostep.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Olostep + Dify

> Give Dify agents, chatflows and workflows live web search, scraping, crawling and cited answers with the official Olostep plugin.

The Olostep plugin brings web data to [Dify](https://dify.ai) agents, chatflows and workflows. [Olostep](https://www.olostep.com) is web data infrastructure for AI agents and enterprise AI workflows. Its APIs provide live web search, scraping and crawling to extract, clean and structure data from the web.

[Install from the Dify Marketplace →](https://marketplace.dify.ai/plugin/olostep/olostep)

## Why Olostep

* Web data infrastructure for developers and enterprises
* Scales to hundreds of millions to billions of requests per customer per month
* The most performant, reliable and cost-effective API on the market
* Used by the fastest-growing AI companies

From extracting data from a handful of URLs to crawling hundreds of millions of pages, Olostep gives developers and AI companies reliable, scalable and highly cost-effective access to clean, structured web data.

## Tools

| Tool | What it does |
| - | - |
| Scrape URL | Scrape a page and return clean markdown (JavaScript rendering included) |
| Search Web | Search the live web and return titles, URLs and descriptions |
| Answer Question | Answer a question from the live web, with source URLs |
| Map Website | List the URLs on a website (up to 100k) |
| Crawl Website | Crawl a website and return the markdown of each page |
| Batch Scrape | Scrape up to 100k URLs in one batch and return the markdown of each page |
| Get Results | Get the pages of a crawl or batch, up to 100 per call |

## Setup

1. Sign up at [olostep.com](https://www.olostep.com) and copy your API key from the [Olostep dashboard](https://www.olostep.com/dashboard).
2. In Dify, go to **Plugins**, find **Olostep** in the Marketplace and click **Install**.
3. Open the plugin, click **Authorize** and paste your API key.

Dify stores the key as a secret. The plugin sends it only to the Olostep API.

## Usage

### Agents

Add the Olostep tools to an agent app. The agent picks the tool it needs, for example Search Web to find pages and Scrape URL to read them.

### Workflows and chatflows

Add a tool node, choose an Olostep tool and map its inputs. Every tool returns JSON for later nodes and readable text for LLM nodes.

For example, a research workflow can run **Search Web** for a topic, then **Batch Scrape** on the result URLs, then summarize the pages with an LLM node.

## Crawls and batches

Crawl Website and Batch Scrape wait for results for up to 2 minutes by default. You can raise this to 4 minutes with **Wait for results (seconds)**.

If the job is still running when the wait ends, the tool returns its `id` with `status` set to `in_progress`. Pass that `id` to **Get Results** later to get the pages.

Each call returns up to 100 pages. If more are available, the result includes `next_cursor`: pass it to **Get Results** with the `id` to get the next pages.

> [!NOTE]
>
> Batch Scrape takes up to 100k URLs. New accounts are limited to 100 URLs per batch: contact [info@olostep.com](mailto:info@olostep.com) to lift this.

## Self-hosted Dify

The plugin connects only to `https://api.olostep.com` over HTTPS, so your Dify instance needs outbound network access to that host.

## Resources

* [Dify Marketplace listing](https://marketplace.dify.ai/plugin/olostep/olostep)
* [GitHub repository](https://github.com/olostep-api/olostep-dify-plugin)
* [Dify documentation](https://docs.dify.ai)
* [Olostep website](https://www.olostep.com)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.