> ## Documentation Index
> Fetch the complete documentation index at: https://docs.olostep.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Welcome to Olostep

> Olostep: Infrastructure for the Web's second user. The best search, scraping and crawling API for AI.

Olostep is infrastructure for the Web's second user — giving AI agents a way to search the web, extract structured data in real time, and build custom research agents.

<img src="https://mintcdn.com/olostep-58/ndWPtiE100GTiImQ/images/explainer_v5.png?fit=max&auto=format&n=ndWPtiE100GTiImQ&q=85&s=ae92c6f02390b885a46f830090b0de34" alt="Main dashboard interface" style={{ width: '100%', height: 'auto', maxHeight: 420, objectFit: 'contain' }} className="rounded-lg" width="2174" height="940" data-path="images/explainer_v5.png" />

## Introduction

* The **Olostep API** is the best **web search**, **scraping** and **crawling** API for AI used by some of the leading startups and scaleups in the world.

* The **Olostep Agent** allows to automate **research workflows** in a no code way with just a prompt in natural language.

<Note>
  **For AI agents:** fetch [docs.olostep.com/llms.txt](https://docs.olostep.com/llms.txt) for a complete index of this documentation before exploring further or [get started here](https://www.olostep.com/agent-onboarding/SKILL.md).
</Note>

## Use Olostep from your terminal and AI agents

Beyond the API, Olostep ships a CLI, an MCP server, and drop-in skills so any tool — Claude Code, Cursor, Windsurf, and more — can use the web natively.

<CardGroup cols={3}>
  <Card title="CLI →" icon="terminal" href="/sdks/cli">
    `npm i -g olostep-cli` — scrape, map, crawl, answer, and batch the web from your terminal. JSON output for scripts, CI, and agents.
  </Card>

  <Card title="MCP Server →" icon="plug" href="/integrations/mcp-server">
    Give any MCP client (Claude, Cursor, VS Code) live web tools. Hosted endpoint — no install.
  </Card>

  <Card title="Skills →" icon="sparkles" href="/features/skills">
    Drop-in skills that teach AI coding agents how and when to use Olostep. Install with `olostep add skills`.
  </Card>
</CardGroup>

## What can Olostep do?

<CardGroup cols={3}>
  <Card title="Scrape" icon="file-lines" href="#scrape">
    Pull any URL as clean Markdown, HTML, screenshots, or structured JSON.
  </Card>

  <Card title="Crawl" icon="spider" href="#crawl">
    Recursively gather every page on a site, with filters and search.
  </Card>

  <Card title="Answer" icon="comments-question" href="#answer">
    Get AI-synthesised answers from live web sources, with citations.
  </Card>
</CardGroup>

### Why Olostep?

* **Built for AI**: Clean Markdown, structured JSON, citations — output your agents and apps consume directly.
* **Reliable at scale**: Industry-leading success rate; handles JavaScript, anti-bot, and proxies under the hood.
* **Fast**: Sub-second single scrape; up to 10,000 URLs in a single batch in 5–7 minutes.
* **Cost-effective**: Significantly cheaper than alternatives at production scale.
* **CLI + MCP + Skills**: Use Olostep from your terminal, scripts, or any MCP-aware agent — agent skills included.

***

## Scrape

Pull any URL as clean Markdown. See the [Scrape feature docs](/features/scrapes) for all options.

<CodeGroup>
  ```python Python theme={null}
  from olostep import Olostep

  client = Olostep(api_key="YOUR_REAL_KEY")
  result = client.scrapes.create(
      url_to_scrape="https://en.wikipedia.org/wiki/Alexander_the_Great",
      formats=["markdown"],
  )
  print(result.markdown_content)
  ```

  ```js Node theme={null}
  import Olostep from 'olostep'

  const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
  const result = await client.scrapes.create({
    url: 'https://en.wikipedia.org/wiki/Alexander_the_Great',
    formats: ['markdown'],
  })
  console.log(result.markdown_content)
  ```

  ```bash cURL theme={null}
  curl -s -X POST "https://api.olostep.com/v1/scrapes" \
    -H "Authorization: Bearer $OLOSTEP_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url_to_scrape": "https://en.wikipedia.org/wiki/Alexander_the_Great",
      "formats": ["markdown"]
    }'
  ```

  ```bash CLI theme={null}
  olostep scrape "https://en.wikipedia.org/wiki/Alexander_the_Great"
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "id": "scrape_6h89o8u1kt",
    "object": "scrape",
    "result": {
      "markdown_content": "## Alexander the Great...",
      "markdown_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/markDown_6h89o8u1kt.txt",
      "page_metadata": { "status_code": 200, "title": "Alexander the Great - Wikipedia" }
    }
  }
  ```
</Accordion>

## Crawl

Recursively gather every page on a site, with include/exclude filters and an optional `search_query` to focus the crawl. See [Crawl feature docs](/features/crawls).

<CodeGroup>
  ```python Python theme={null}
  from olostep import Olostep

  client = Olostep(api_key="YOUR_REAL_KEY")
  crawl = client.crawls.create(
      start_url="https://docs.olostep.com",
      max_pages=50,
  )
  for page in crawl.pages():
      print(page.url)
  ```

  ```js Node theme={null}
  import Olostep from 'olostep'

  const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
  const crawl = await client.crawls.create({
    url: 'https://docs.olostep.com',
    maxPages: 50,
  })
  for await (const page of crawl.pages()) {
    console.log(page.url)
  }
  ```

  ```bash cURL theme={null}
  curl -s -X POST "https://api.olostep.com/v1/crawls" \
    -H "Authorization: Bearer $OLOSTEP_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "start_url": "https://docs.olostep.com",
      "max_pages": 50
    }'
  ```

  ```bash CLI theme={null}
  olostep crawl "https://docs.olostep.com" --max-pages 50
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "id": "crawl_abc123",
    "object": "crawl",
    "status": "completed",
    "pages_count": 47,
    "pages": [
      { "url": "https://docs.olostep.com/get-started/welcome", "retrieve_id": "..." },
      { "url": "https://docs.olostep.com/features/scrapes", "retrieve_id": "..." }
    ]
  }
  ```
</Accordion>

## Answer

Ask a question and get an AI-synthesised answer from live web sources, with citations. Pass a JSON schema to shape the output. See [Answers docs](/features/answers).

<CodeGroup>
  ```python Python theme={null}
  from olostep import Olostep

  client = Olostep(api_key="YOUR_REAL_KEY")
  answer = client.answers.create(task="What does Olostep do?")
  print(answer.result)
  print(answer.sources)
  ```

  ```js Node theme={null}
  import Olostep from 'olostep'

  const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
  const answer = await client.answers.create({ task: 'What does Olostep do?' })
  console.log(answer.result)
  console.log(answer.sources)
  ```

  ```bash cURL theme={null}
  curl -s -X POST "https://api.olostep.com/v1/answers" \
    -H "Authorization: Bearer $OLOSTEP_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"task": "What does Olostep do?"}'
  ```

  ```bash CLI theme={null}
  olostep answer "What does Olostep do?"
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "id": "answer_abc123",
    "object": "answer",
    "task": "What does Olostep do?",
    "result": {
      "json_content": "{\"result\":\"Olostep is an API that lets AI agents search, scrape, and structure web data.\"}",
      "sources": [
        "https://docs.olostep.com/get-started/welcome",
        "https://www.olostep.com/"
      ]
    }
  }
  ```
</Accordion>

***

## More capabilities

<CardGroup cols={2}>
  <Card title="Batch" icon="boxes-stacked" href="/features/batches">
    Scrape up to 10,000 URLs in parallel; results back in 5–7 minutes.
  </Card>

  <Card title="Map" icon="map" href="/features/maps">
    Discover every URL on a site with include/exclude patterns.
  </Card>

  <Card title="Search" icon="magnifying-glass" href="/features/search">
    Live web search with structured links and optional inline scraping.
  </Card>

  <Card title="Parsers" icon="brackets-curly" href="/features/structured-content/parsers">
    Self-healing extractors that turn pages into typed JSON at scale.
  </Card>

  <Card title="Schedules" icon="calendar" href="/features/schedules">
    Run scrapes, crawls, and answers on a recurring schedule.
  </Card>

  <Card title="Files" icon="file" href="/features/files">
    Upload files for batches or to connect your knowledge base.
  </Card>
</CardGroup>

***

## Resources

<CardGroup cols={2}>
  <Card title="Explore Features" icon="graduation-cap" href="/features/scrapes">
    Check out all supported features for your scraping and AI search needs.
  </Card>

  <Card title="API Reference" icon="comment" href="/api-reference/scrapes/create">
    Start using the API and test out the various params.
  </Card>

  <Card title="Integrations" icon="plug" href="/integrations/n8n">
    Use Olostep in n8n, make, relay, zapier, etc
  </Card>

  <Card title="Examples" icon="code" href="/examples/">
    Browse ready-to-use examples to get started quickly.
  </Card>
</CardGroup>