> ## Documentation Index
> Fetch the complete documentation index at: https://docs.olostep.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Olostep CLI

> Command-line interface for scrape, search, map, crawl, answers, and batches — JSON output for scripts, CI, and AI agents

**NPM package:** [olostep-cli](https://www.npmjs.com/package/olostep-cli)
**Repository:** [github.com/olostep-api/olostep-cli](https://github.com/olostep-api/olostep-cli)

CLI for the [Olostep API](https://www.olostep.com/) — **scrape**, **search**, **map**, **crawl**, **answer**, and **batch** the web from your terminal. Every command returns **JSON** so it pipes cleanly into `jq`, agents, and CI.

Pure JavaScript, Node 18+, no native binaries to download. Installs in under a second, starts in \~200 ms, ships as a single \~100 KB bundle.

## Install

**Requirements:** Node.js **18+**.

<Tabs>
  <Tab title="npm (recommended)">
    ```bash theme={null}
    npm install -g olostep-cli
    olostep init
    ```
  </Tab>

  <Tab title="One-liner (macOS / Linux)">
    ```bash theme={null}
    curl -fsSL https://olostep.com/install.sh | sh
    olostep init
    ```

    The script checks Node 18+, runs `npm install -g olostep-cli`, and falls back to `sudo` if needed.
  </Tab>

  <Tab title="One-liner (Windows)">
    ```powershell theme={null}
    iwr -useb https://olostep.com/install.ps1 | iex
    olostep init
    ```

    Same idea as the macOS/Linux script, but for PowerShell.
  </Tab>

  <Tab title="No-install (npx)">
    ```bash theme={null}
    npx -y olostep-cli@latest --help
    ```

    Good for trying a single command without a global install.
  </Tab>
</Tabs>

`olostep init` is the recommended next step — it signs you in, installs the Olostep skills into your AI agents, and configures the MCP server, all in one command. The one-liner scripts wrap `npm install -g olostep-cli` with a Node 18+ check and a `sudo` fallback, so they work even if you're unsure of your local setup.

**Platforms:** macOS (Apple Silicon and Intel), Linux (x64 and arm64), Windows (x64 and arm64).

## Set up

One command does everything — sign in, install skills, and install the MCP server:

```bash theme={null}
olostep init
```

Flags: `--skills-only`, `--mcp-only`, `--no-browser`, `--relogin`.

To **just sign in** (no skills/MCP):

```bash theme={null}
olostep login
olostep login --no-browser     # print the URL (useful over SSH)
```

The browser opens to the Olostep auth page; you click **Authorize**, and the CLI saves your key locally.

**Alternative — set an env var.** Good for CI:

```bash theme={null}
export OLOSTEP_API_KEY=your_key_here
```

Get a key from the [API Keys dashboard](https://www.olostep.com/dashboard/api-keys).

**Where the key is stored** (after `olostep login`):

| OS      | Path                                                         |
| ------- | ------------------------------------------------------------ |
| macOS   | `~/Library/Application Support/olostep-cli/credentials.json` |
| Linux   | `~/.config/olostep-cli/credentials.json`                     |
| Windows | `%USERPROFILE%\AppData\Roaming\olostep-cli\credentials.json` |

## Sign out

```bash theme={null}
olostep logout            # prompts to confirm, then removes credentials.json
olostep logout --dry-run  # preview only — see what would happen
olostep logout --yes      # skip the confirmation (for scripts)
olostep logout --json     # machine-readable output
```

`logout` also warns you if `OLOSTEP_API_KEY` / `OLOSTEP_API_TOKEN` env vars or a `.env` file in your current directory still hold a key — those take priority over the credentials file, so deleting the file alone may not be enough. The output includes the exact unset commands for PowerShell and bash/zsh.

## Quick start

```bash theme={null}
olostep login

olostep search "best web scraping APIs 2025" --limit 5
olostep answer "What does Olostep do?"
olostep map "https://example.com" --top-n 20
olostep scrape "https://example.com" --formats markdown
olostep crawl "https://docs.example.com" --max-pages 50
olostep batch-scrape urls.csv --formats markdown,html
```

Every command prints its JSON result to stdout by default. Pass `--out <path>` to save to a file.

## What can it do?

| You want to…              | Command                         | Olostep product                                 |
| ------------------------- | ------------------------------- | ----------------------------------------------- |
| Search the web            | `search`                        | [Searches](/features/search)                    |
| Get a researched answer   | `answer`                        | [Answers](/features/answers)                    |
| Discover URLs on a site   | `map`                           | [Maps](/features/maps)                          |
| Pull one page             | `scrape`                        | [Scrapes](/features/scrapes)                    |
| Pull every page on a site | `crawl`                         | [Crawls](/features/crawls)                      |
| Pull many URLs from a CSV | `batch-scrape`                  | [Batches](/features/batches)                    |
| Extract structured fields | `--parser-id` on `batch-scrape` | [Parsers](/features/structured-content/parsers) |
| Refetch a result by ID    | `scrape-get`                    | [Scrapes](/features/scrapes)                    |
| Tag/organize a batch      | `batch-update`                  | [Batches](/features/batches)                    |

## Output

Every command **prints its JSON result to stdout** by default.

| Flag           | Behavior                                   |
| -------------- | ------------------------------------------ |
| *(none)*       | Print JSON to **stdout** (UTF-8, indented) |
| `--out <path>` | Write JSON to that file instead            |
| `--out -`      | Explicitly stdout (same as default)        |

Progress and log lines go to **stderr**, so stdout stays clean for pipes.

```bash theme={null}
olostep map "https://example.com" --top-n 20 | jq '.urls[:10]'
olostep scrape "https://example.com" | jq .result.markdown_content
olostep search "topic" --json | jq '.links[].url'
```

**Choosing between them:**

* **`search`** — you want a list of relevant URLs and snippets for a query. The CLI searches the web for you.
* **`answer`** — you want a synthesized answer, not raw page content. The CLI does the research for you.
* **`scrape`** — you already have the URL and want clean content out.
* **`crawl`** — you want every page on a site (or a filtered subset) without enumerating URLs by hand.
* **`batch-scrape`** — you have a list of URLs and want them processed in parallel.

## Commands

Use `olostep <command> --help` for every option.

### `search`: live web search

Returns deduplicated organic links (URL, title, description).

| Option              | Description                                    |
| ------------------- | ---------------------------------------------- |
| `--limit`           | Number of results, default 12, max 25          |
| `--include-domains` | Comma-separated domains to restrict results to |
| `--exclude-domains` | Comma-separated domains to exclude             |
| `--out`             | File or `-`                                    |
| `--json`            | Machine-readable output                        |

```bash theme={null}
olostep search "TypeScript CLI tools" --limit 10
olostep search "open source projects" --include-domains "github.com" --limit 5
olostep search "AI agents" --json | jq '.links[].url'
```

### `answer`: researched answer

Synchronous — returns when the answer is ready.

| Option          | Description                               |
| --------------- | ----------------------------------------- |
| `--out`         | File or `-`                               |
| `--json-format` | Optional JSON shape for structured output |

```bash theme={null}
olostep answer "What does this company build?" --out answer.json
olostep answer "Extract facts" --json-format '{"company":"","year":""}' --out -
```

### `map`: discover URLs

| Option                                           | Description                       |
| ------------------------------------------------ | --------------------------------- |
| `--out`                                          | File path or `-`                  |
| `--top-n`                                        | Max URLs to return                |
| `--search-query`                                 | Optional query to guide discovery |
| `--include-subdomain` / `--no-include-subdomain` | Subdomains                        |
| `--include-url` / `--exclude-url`                | Repeatable URL patterns           |
| `--cursor`                                       | Pagination cursor                 |

```bash theme={null}
olostep map "https://example.com" --top-n 100 --search-query "blog"
```

### `scrape`: one URL

**Formats:** `html`, `markdown`, `text`, `json`, `raw_pdf`, `screenshot` (comma-separated; default `markdown`).

| Option                              | Description                                                                                                       |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `--formats`                         | Comma-separated                                                                                                   |
| `--country`                         | Country code (e.g. `US`, `GB`)                                                                                    |
| `--wait-before-scraping`            | Wait before scrape (ms)                                                                                           |
| `--payload-json` / `--payload-file` | Advanced options as JSON (e.g. `"max_age": 86400` to opt into caching — see [Caching](/features/scrapes#caching)) |

```bash theme={null}
olostep scrape "https://example.com" --formats markdown,html
olostep scrape "https://example.com" --payload-file options.json --out -
```

### `scrape-get`: fetch by ID

```bash theme={null}
olostep scrape-get "scrape_abc123" --out -
```

### `crawl`: whole site

Starts a crawl, polls until finished, then retrieves page contents.

**Retrieve formats:** `markdown`, `html`, `json`.

Notable flags: `--max-pages`, `--max-depth`, `--include-subdomain`, `--include-external`, `--include-url`, `--exclude-url`, `--search-query`, `--top-n`, `--webhook`, `--crawl-timeout`, `--formats`, `--pages-limit`, `--pages-search-query`, `--poll-seconds`, `--poll-timeout`, `--dry-run`.

```bash theme={null}
olostep crawl "https://docs.example.com" --max-pages 50 --formats markdown,html
olostep crawl "https://example.com" --max-pages 10 --dry-run
```

### `batch-scrape`: CSV

CSV must have a header row with **`custom_id`** (or `id`) and **`url`** columns.

```csv theme={null}
custom_id,url
example,https://example.com
iana,https://iana.org
docs,https://docs.olostep.com
```

| Option                                           | Description                                  |
| ------------------------------------------------ | -------------------------------------------- |
| `--formats`                                      | `markdown`, `html`, `json` (comma-separated) |
| `--country`                                      | Optional country code                        |
| `--parser-id`                                    | Parser ID for structured extraction          |
| `--poll-seconds`, `--log-every`, `--items-limit` | Polling and paging                           |
| `--dry-run`                                      | Print payload and exit                       |

```bash theme={null}
olostep batch-scrape urls.csv --formats markdown,html
olostep batch-scrape urls.csv --parser-id "<PARSER_ID>" --out results.json
```

Synchronous — polls until the batch completes, then retrieves every item.

### `batch-update`: batch metadata

Requires **one of** `--metadata-json` or `--metadata-file` (JSON object).

```bash theme={null}
olostep batch-update "batch_abc123" --metadata-json '{"team":"growth"}'
olostep batch-update "batch_abc123" --metadata-file meta.json
```

## Auth commands

```bash theme={null}
olostep login                      # browser PKCE sign-in
olostep logout                     # remove saved credentials
olostep status                     # show auth state, config paths, version

olostep auth login                 # same as olostep login
olostep auth logout                # same as olostep logout
olostep auth status                # same as olostep status
olostep auth set-key <key>         # save an API key directly (no browser)
```

`auth set-key` is useful for CI and scripts — write the key directly without going through the browser flow.

## Install the MCP server

The CLI writes the Olostep MCP server into your agent's config — no JSON editing.

```bash theme={null}
olostep mcp install                          # detect agents, hosted endpoint
olostep mcp install --agent cursor           # only Cursor
olostep mcp install --transport stdio        # local npx instead of hosted
olostep mcp install --no-global              # write into current project
olostep mcp install --dry-run --json         # plan only
olostep mcp uninstall                        # remove the olostep entry
olostep list mcp                             # show which agents have it
```

| Option                             | Description                                                                                                                                |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `--agent`                          | Specific agent, repeatable. Supported: `cursor`, `claude`, `claude-desktop`, `windsurf`, `vscode`, `kilo`, `opencode`, `continue`, `codex` |
| `--all-agents` / `--no-all-agents` | Target every detected agent (default)                                                                                                      |
| `--transport`                      | `http` (hosted, recommended) or `stdio` (local `npx olostep-mcp`)                                                                          |
| `--global` / `--no-global`         | Per-user config (default) vs project-local                                                                                                 |
| `--api-key`                        | Key to embed; defaults to resolved credentials                                                                                             |
| `--dry-run`                        | Show the plan without writing                                                                                                              |
| `--json`                           | Machine-readable output                                                                                                                    |

The hosted endpoint at `https://mcp.olostep.com/mcp` uses `Authorization: Bearer <key>` — no local Node process required. The CLI merges only the `olostep` key into your existing config. Restart your agent after install.

## Skills for AI agents

The CLI ships **13 Olostep skills** — `SKILL.md` files installed into Claude Code, Cursor, and other agents so they know what Olostep can do and when to use it.

```bash theme={null}
olostep add skills                     # install all into every detected agent
olostep skills install                 # same (alias)
olostep skills update                  # re-install / refresh all skills
olostep skills list                    # see what's installed and where
olostep skills uninstall               # remove all skills
```

Filter what gets installed:

```bash theme={null}
olostep add skills --category usage    # core web-data skills only
olostep add skills --skill scrape --skill map
olostep add skills --agent cursor --agent claude
```

See [Skills](/features/skills) for the full list and options.

## Health checks

```bash theme={null}
olostep doctor                         # run all checks
olostep doctor --skip-network          # auth + config only, no HTTP calls
olostep doctor --json                  # NDJSON — one record per check (good for CI)
olostep doctor --fail-on-warn          # exit 1 on warnings too
```

Checks: API key present, API key reachable, MCP endpoint reachable, config file exists for each detected agent.

CI usage:

```bash theme={null}
olostep doctor --json --skip-network | jq 'select(.status == "fail")'
```

## Version & updates

```bash theme={null}
olostep version                        # CLI version, Node version, channel
olostep version --json                 # machine-readable: { cli, node, channel }
olostep update                         # update to latest (npm install -g olostep-cli@latest)
olostep update --check                 # check whether a newer version is available, don't install
```

## Environment variables

| Variable                    | Effect                                                         |
| --------------------------- | -------------------------------------------------------------- |
| `OLOSTEP_API_KEY`           | API key                                                        |
| `OLOSTEP_API_TOKEN`         | API key (legacy alias)                                         |
| `OLOSTEP_JSON=1`            | Force JSON output on every command (same as `--json` globally) |
| `OLOSTEP_NO_UPDATE_CHECK=1` | Silence the "update available" notice                          |
| `OLOSTEP_CLI_CONFIG_DIR`    | Override the credentials directory                             |

## Windows / PowerShell notes

PowerShell tokenizes `,` and `*` differently from bash — quote arguments:

```powershell theme={null}
olostep scrape "https://example.com" --formats "markdown,html"
olostep map   "https://example.com" --include-url "/*"
olostep answer "Extract facts" --json-format '{"company":"","year":""}'
```

Single quotes are safest for JSON values (no `$` interpolation).

## See what's installed

```bash theme={null}
olostep list skills    # installed Olostep skills and which agents have them
olostep list mcp       # which agents have the Olostep MCP server, and the transport
```

## Global flags

| Flag              | Description |
| ----------------- | ----------- |
| `-V`, `--version` | Version     |
| `-h`, `--help`    | Help        |

`--out`, `--timeout`, and `--api-key` are available on every data command.

## Security

Keep API keys out of source control; rotate if leaked. `olostep logout` removes the local credentials file and tells you if any env-var sources still hold a key.

## Related

* [Maps](/features/maps) · [Crawls](/features/crawls) · [Batches](/features/batches) · [Answers](/features/answers) · [Searches](/features/search) · [Skills](/features/skills) · [MCP Server](/integrations/mcp-server)