> ## Documentation Index
> Fetch the complete documentation index at: https://docs.olostep.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Olostep MCP Server

> Give any MCP-compatible AI client web scraping, search, crawling, and AI-answer tools in under a minute

The Olostep MCP server gives any MCP-compatible AI client (Claude, Cursor, Windsurf, VS Code, Claude Code, etc.) 10 ready-to-use tools for the live web — scraping, search, AI answers with citations, batch jobs, site crawling, and URL discovery.

<CardGroup cols={2}>
  <Card title="Scrape & extract" icon="file-lines">
    Pull markdown, HTML, JSON, or text from any URL with optional JS rendering
  </Card>

  <Card title="AI answers" icon="sparkles">
    Web-grounded answers with sources and structured output
  </Card>

  <Card title="Batch & crawl" icon="layer-group">
    Up to 10k URLs in parallel, or autonomously discover a whole site
  </Card>

  <Card title="Map & search" icon="map">
    Find every URL on a site, or run parser-based web search
  </Card>
</CardGroup>

## Before you start

You need an Olostep API key. Get one from the [Olostep dashboard](https://www.olostep.com/dashboard/api-keys) — the free tier covers personal use.

## Pick a setup path

The fastest path for every client is the **hosted endpoint** at `https://mcp.olostep.com/mcp`. No installs, no Node, no Docker — just paste a URL and your API key.

If you need it to run fully local (offline use, corporate proxy, air-gapped), every client also supports a **local stdio** install via `npx`. Each section below shows both.

<Note>
  **Hosted endpoint** uses `Authorization: Bearer YOUR_API_KEY`. **Local stdio** uses `OLOSTEP_API_KEY` as an environment variable. Don't mix them up — wrong auth mode is the #1 onboarding error.
</Note>

## Client setup

<Tabs>
  <Tab title="Cursor">
    **One-click install (recommended):**

    <a href="cursor://anysphere.cursor-deeplink/mcp/install?name=olostep&config=eyJ1cmwiOiJodHRwczovL21jcC5vbG9zdGVwLmNvbS9tY3AiLCJoZWFkZXJzIjp7IkF1dGhvcml6YXRpb24iOiJCZWFyZXIgWU9VUl9BUElfS0VZIn19">
      <img src="https://cursor.com/deeplink/mcp-install-dark.png" alt="Add Olostep MCP server to Cursor" style={{ maxHeight: 32 }} />
    </a>

    Replace `YOUR_API_KEY` in the resulting config with your real key.

    **Manual setup:**

    Create or edit `.cursor/mcp.json` in your project root (or `~/.cursor/mcp.json` for global):

    ```json theme={null}
    {
      "mcpServers": {
        "olostep": {
          "url": "https://mcp.olostep.com/mcp",
          "headers": {
            "Authorization": "Bearer YOUR_API_KEY"
          }
        }
      }
    }
    ```

    <Accordion title="Local stdio install (optional)">
      ```json theme={null}
      {
        "mcpServers": {
          "olostep": {
            "command": "npx",
            "args": ["-y", "olostep-mcp"],
            "env": {
              "OLOSTEP_API_KEY": "YOUR_API_KEY"
            }
          }
        }
      }
      ```

      Requires Node.js 18+ on your machine.
    </Accordion>

    **Verify:** Open Cursor → Settings → MCP. You should see `olostep` listed with **10 tools** including `scrape_website`. If you see "Connected, 0 tools", your API key is wrong.
  </Tab>

  <Tab title="Claude Code">
    **CLI install (recommended):**

    ```bash theme={null}
    claude mcp add --transport http olostep https://mcp.olostep.com/mcp \
      --header "Authorization: Bearer YOUR_API_KEY"
    ```

    **Manual setup:**

    Add to your Claude Code MCP config (`.mcp.json` in project root, or `~/.claude.json` globally):

    ```json theme={null}
    {
      "mcpServers": {
        "olostep": {
          "url": "https://mcp.olostep.com/mcp",
          "headers": {
            "Authorization": "Bearer YOUR_API_KEY"
          }
        }
      }
    }
    ```

    <Accordion title="Local stdio install (optional)">
      ```bash theme={null}
      claude mcp add --transport stdio --env OLOSTEP_API_KEY=YOUR_API_KEY olostep \
        -- npx -y olostep-mcp
      ```

      Or as JSON:

      ```json theme={null}
      {
        "mcpServers": {
          "olostep": {
            "command": "npx",
            "args": ["-y", "olostep-mcp"],
            "env": {
              "OLOSTEP_API_KEY": "YOUR_API_KEY"
            }
          }
        }
      }
      ```
    </Accordion>

    **Verify:** Run `/mcp` in Claude Code. You should see `olostep` connected with 10 tools.
  </Tab>

  <Tab title="Claude Desktop">
    **Config file location:**

    | OS      | Path                                                              |
    | ------- | ----------------------------------------------------------------- |
    | macOS   | `~/Library/Application Support/Claude/claude_desktop_config.json` |
    | Windows | `%APPDATA%\Claude\claude_desktop_config.json`                     |
    | Linux   | `~/.config/Claude/claude_desktop_config.json`                     |

    **Hosted (recommended):**

    ```json theme={null}
    {
      "mcpServers": {
        "olostep": {
          "url": "https://mcp.olostep.com/mcp",
          "headers": {
            "Authorization": "Bearer YOUR_API_KEY"
          }
        }
      }
    }
    ```

    <Accordion title="Local stdio install (optional)">
      ```json theme={null}
      {
        "mcpServers": {
          "olostep": {
            "command": "npx",
            "args": ["-y", "olostep-mcp"],
            "env": {
              "OLOSTEP_API_KEY": "YOUR_API_KEY"
            }
          }
        }
      }
      ```

      Or install via Smithery:

      ```bash theme={null}
      npx -y @smithery/cli install @olostep/olostep-mcp-server --client claude
      ```
    </Accordion>

    <Warning>
      Claude Desktop must be **fully quit and relaunched** for config changes to take effect — closing the window isn't enough (it stays running in the menu bar / system tray).
    </Warning>

    **Verify:** Open Claude Desktop → look for the 🔨 (hammer) icon in the chat input. Click it — you should see 10 Olostep tools listed.
  </Tab>

  <Tab title="VS Code">
    VS Code's MCP support is built into GitHub Copilot (Agent mode). Add this to `.vscode/mcp.json` in your project, or your user `settings.json`:

    ```json theme={null}
    {
      "servers": {
        "olostep": {
          "type": "http",
          "url": "https://mcp.olostep.com/mcp",
          "headers": {
            "Authorization": "Bearer YOUR_API_KEY"
          }
        }
      }
    }
    ```

    <Accordion title="Local stdio install (optional)">
      ```json theme={null}
      {
        "servers": {
          "olostep": {
            "type": "stdio",
            "command": "npx",
            "args": ["-y", "olostep-mcp"],
            "env": {
              "OLOSTEP_API_KEY": "YOUR_API_KEY"
            }
          }
        }
      }
      ```
    </Accordion>

    **Verify:** Open the Copilot chat panel → switch to Agent mode → the tools popover should list Olostep tools.
  </Tab>

  <Tab title="Windsurf">
    Add to `~/.codeium/windsurf/mcp_config.json`:

    ```json theme={null}
    {
      "mcpServers": {
        "olostep": {
          "serverUrl": "https://mcp.olostep.com/mcp",
          "headers": {
            "Authorization": "Bearer YOUR_API_KEY"
          }
        }
      }
    }
    ```

    <Accordion title="Local stdio install (optional)">
      ```json theme={null}
      {
        "mcpServers": {
          "olostep": {
            "command": "npx",
            "args": ["-y", "olostep-mcp"],
            "env": {
              "OLOSTEP_API_KEY": "YOUR_API_KEY"
            }
          }
        }
      }
      ```
    </Accordion>

    **Verify:** Cascade → Settings → MCP. `olostep` should appear with 10 tools.
  </Tab>

  <Tab title="Docker">
    If you'd rather run the server in a container (CI, isolated env, no Node on host):

    ```bash theme={null}
    docker pull olostep/mcp-server

    docker run -i --rm \
      -e OLOSTEP_API_KEY="YOUR_API_KEY" \
      olostep/mcp-server
    ```

    In an MCP client config (stdio):

    ```json theme={null}
    {
      "mcpServers": {
        "olostep": {
          "command": "docker",
          "args": [
            "run", "-i", "--rm",
            "-e", "OLOSTEP_API_KEY=YOUR_API_KEY",
            "olostep/mcp-server"
          ]
        }
      }
    }
    ```

    Supports `linux/amd64` and `linux/arm64`. Source on [GitHub](https://github.com/olostep/olostep-mcp-server).
  </Tab>

  <Tab title="Metorial">
    1. Open the [Metorial dashboard](https://metorial.com)
    2. Navigate to **MCP Servers**
    3. Search for **Olostep**
    4. Click **Install** and paste your API key

    For manual configuration:

    ```json theme={null}
    {
      "olostep": {
        "command": "npx",
        "args": ["-y", "olostep-mcp"],
        "env": {
          "OLOSTEP_API_KEY": "YOUR_API_KEY"
        }
      }
    }
    ```
  </Tab>
</Tabs>

## Picking the right tool

The MCP server exposes 10 tools. Use this decision tree to pick the right one — the agent uses the same reasoning:

| You want...                                | Use                                       | Notes                                         |
| ------------------------------------------ | ----------------------------------------- | --------------------------------------------- |
| A specific page's content                  | `scrape_website` or `get_webpage_content` | Set `wait_before_scraping=2000–5000` for SPAs |
| A natural-language web answer with sources | `answers`                                 | Returns AI synthesis + citations              |
| Search results for a query                 | `search_web`                              | Parser-based, non-AI, structured              |
| A list of URLs on a site                   | `create_map`                              | URL discovery only — does NOT scrape          |
| URLs filtered by query                     | `get_website_urls`                        | Ranked by relevance to your `search_query`    |
| Many known URLs at once                    | `batch_scrape_urls` + `get_batch_results` | Async — kicks off, then poll                  |
| A whole site or section                    | `create_crawl` + `get_crawl_results`      | Async — follows links from a start URL        |

<Tip>
  **Scraping a whole site?** Use `create_crawl`, not `batch_scrape_urls`. Crawl discovers AND scrapes. Batch is for a known list of URLs you already have.
</Tip>

### Tool details

<Accordion title="scrape_website">
  Extract content from a single URL. Supports `markdown`, `html`, `json`, `text`. Optional `country` for geo-targeted requests, `wait_before_scraping` (0–10000 ms) for JS-heavy sites, and `parser` (e.g. `@olostep/amazon-product`) for structured extraction.
</Accordion>

<Accordion title="get_webpage_content">
  Lightweight markdown-only version of `scrape_website`. Use when you just want clean markdown and don't need format options.
</Accordion>

<Accordion title="search_web">
  Structured (parser-based) web search results for a query. Optional `country` for localized results. Returns JSON, not AI prose.
</Accordion>

<Accordion title="answers">
  AI-powered answer to a `task` with sources and citations. Pass a `json` argument to get the answer in a specific shape — either a JSON schema or a short natural-language description.
</Accordion>

<Accordion title="batch_scrape_urls">
  Async scrape of 2–10k URLs you already have. Returns a `batch_id` — then call `get_batch_results` to fetch content. Set `wait_for_completion_seconds` (up to 900) if you want a single blocking call instead of polling. Recommended: 60 for batches under 50 URLs, 300–600 for 50–1k, 0 (poll separately) for larger batches.
</Accordion>

<Accordion title="get_batch_results">
  Fetches the status and scraped content for a `batch_id`. Returns `processing` until done, then `completed` with the items array.
</Accordion>

<Accordion title="create_crawl">
  Async crawl that follows links from a `start_url`. Use `include_url_patterns` / `exclude_url_patterns` (glob syntax like `/blog/**`) to scope. Returns a `crawl_id` — then call `get_crawl_results`.
</Accordion>

<Accordion title="get_crawl_results">
  Fetches the status and pages for a `crawl_id`. Supports pagination via `cursor` and `items_limit` (max 100 per call). Returns `in_progress` until done.
</Accordion>

<Accordion title="create_map">
  Get a list of URLs on a site. URL discovery only — does not scrape. Use when you want to surface candidate URLs (e.g. let the user pick a subset). Supports `include_url_patterns` / `exclude_url_patterns` and `search_query`.
</Accordion>

<Accordion title="get_website_urls">
  Like `create_map`, but URLs are ranked by relevance to a required `search_query`. Use when you want the top N matching links on a site.
</Accordion>

## Troubleshooting

<Accordion title="Server appears but shows 0 tools">
  Your API key is invalid or rate-limited. Open the [API keys dashboard](https://www.olostep.com/dashboard/api-keys) and verify the key. If using the hosted endpoint, the header must be **exactly** `Authorization: Bearer sk_...` — no quotes around the value, no extra spaces.
</Accordion>

<Accordion title="`npx: command not found` or `command not found: olostep-mcp`">
  Node.js isn't installed (or not in your PATH). Install Node 18+ from [nodejs.org](https://nodejs.org/), then restart your terminal **and** your MCP client. On Windows, switch to a CMD/PowerShell that has Node on the PATH.
</Accordion>

<Accordion title="Connection refused or DNS errors on `mcp.olostep.com`">
  You're likely behind a corporate proxy or firewall blocking the host. Switch to the local stdio install (`npx -y olostep-mcp`) — it makes outbound requests to `api.olostep.com` instead, which is usually allowed.
</Accordion>

<Accordion title="Edited config but the tool list is stale">
  The client cached the old config. Fully quit and relaunch — not just close the window. Claude Desktop in particular keeps running in the menu bar / system tray.
</Accordion>

<Accordion title="Windows-specific `npx` failures">
  If `npx` errors out launching the server on Windows, use the CMD-wrapped form:

  ```json theme={null}
  {
    "command": "cmd",
    "args": ["/c", "npx", "-y", "olostep-mcp"],
    "env": { "OLOSTEP_API_KEY": "YOUR_API_KEY" }
  }
  ```
</Accordion>

<Accordion title="`401 Missing Authorization: Bearer <OLOSTEP_API_KEY>`">
  You hit the hosted endpoint without an auth header (or with the wrong format). Add the header to your client config exactly as shown in the setup tab.
</Accordion>

## Recipes

Copy-paste prompts that work well with the tools:

* **Scrape a list of product URLs:** *"I have a CSV of 200 Amazon product URLs. Batch scrape them with `parser=@olostep/amazon-product` and return as JSON."*
* **Crawl a docs site:** *"Crawl [https://stripe.com/docs](https://stripe.com/docs) with `max_pages=50` and `include_url_patterns=['/docs/**']`. Summarize each section as markdown."*
* **Find competitors:** *"Use `answers` to find the top 5 competitors to Notion for technical doc sites. Return name, homepage, and 1-line positioning."*
* **Map then scrape:** *"Run `create_map` on [https://example.com](https://example.com) filtered to `/blog/**`, then `batch_scrape_urls` on the top 20 results."*

## Source & versions

* [GitHub repo](https://github.com/olostep/olostep-mcp-server)
* [npm package](https://www.npmjs.com/package/olostep-mcp)
* [Docker Hub](https://hub.docker.com/r/olostep/mcp-server)
* [MCP Registry](https://registry.modelcontextprotocol.io/)