> ## Documentation Index
> Fetch the complete documentation index at: https://docs.olostep.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Olostep + Vercel AI SDK Integration

> Give AI SDK agents live web search, scraping, crawling and cited answers with ready-made Olostep tools

The Olostep AI SDK integration gives agents built with the [Vercel AI SDK](https://ai-sdk.dev) ready-made tools for live web data. Olostep is a web search, scraping and crawling API: an API to search, extract and structure web data. Pass the tools to `generateText` or `streamText` and the model decides when to search, scrape or crawl.

[Install from npm →](https://www.npmjs.com/package/@olostep/ai-sdk)

## Features

The package provides 6 tools:

> **Scrape**
>
> Scrape any URL and return clean markdown, with JavaScript rendering

> **Search**
>
> Search the live web and return titles, URLs and descriptions

> **Answer**
>
> Answer a question from the live web, with source URLs

> **Map**
>
> List the URLs on a website, optionally ranked by a query

> **Crawl**
>
> Crawl a site from a starting URL and return the markdown of each page

> **Batch**
>
> Scrape a list of URLs in one batch and return the markdown of each

## Installation

```bash npm theme={null}
npm install @olostep/ai-sdk ai zod
```

```bash yarn theme={null}
yarn add @olostep/ai-sdk ai zod
```

```bash pnpm theme={null}
pnpm add @olostep/ai-sdk ai zod
```

Works with AI SDK 5, 6 and 7, Zod 3.25+ or 4, and Node.js 18+.

## Setup

Set your Olostep API key as an environment variable:

```bash theme={null}
export OLOSTEP_API_KEY=your-api-key-here
```

Or in your `.env` file:

```
OLOSTEP_API_KEY=your-api-key-here
```

Get your API key from the [Olostep Dashboard](https://www.olostep.com/dashboard).

## Quick Start

Pass all Olostep tools to the model with `olostepTools()`:

```typescript theme={null}
import { generateText, stepCountIs } from 'ai';
import { olostepTools } from '@olostep/ai-sdk';

const { text } = await generateText({
  model: 'openai/gpt-5-mini',
  tools: olostepTools(),
  stopWhen: stepCountIs(5),
  prompt: 'Read the Next.js 15 release notes and list the breaking changes, with sources.',
});

console.log(text);
```

`stopWhen` lets the model call tools over several steps, for example search first and then scrape the best result. The model string uses the [Vercel AI Gateway](https://vercel.com/docs/ai-gateway). You can also pass any AI SDK provider model, such as `openai('gpt-5-mini')` from `@ai-sdk/openai`.

### Pick Individual Tools

Only give the model the tools it needs:

```typescript theme={null}
import { olostepScrape, olostepSearch } from '@olostep/ai-sdk';

const tools = {
  scrape: olostepScrape(),
  search: olostepSearch(),
};
```

## Available Tools

The tool names below are the keys used by `olostepTools()`.

### scrape

`olostepScrape()`. Scrape a single URL and return its content as markdown.

**Input:**

**url** `string` *required*

The URL of the page to scrape

**waitBeforeScraping** `number`

Milliseconds to wait for JavaScript-heavy pages to render (0 to 10000)

**Returns:** `{ url, markdown }`

### search

`olostepSearch()`. Search the live web.

**Input:**

**query** `string` *required*

The search query

**limit** `number` (default: 10)

Maximum number of results (1 to 20)

**Returns:** `{ query, results: [{ title, url, description }] }`

### answer

`olostepAnswer()`. Search and read the live web to answer a question, with sources.

**Input:**

**task** `string` *required*

The question or research task

**Returns:** `{ answer, sources }`

### map

`olostepMap()`. Discover the URLs on a website.

**Input:**

**url** `string` *required*

The website to map

**searchQuery** `string`

Rank URLs by relevance to this query

**topN** `number`

Maximum number of URLs to return (1 to 500)

**Returns:** `{ url, urls }`

### crawl

`olostepCrawl()`. Crawl a website from a starting URL and return each page's markdown. The tool waits for the crawl to finish before returning.

**Input:**

**url** `string` *required*

The URL to start crawling from

**maxPages** `number` (default: 5)

Maximum pages to crawl (1 to 25)

**includeUrls** `array`

Only crawl URLs matching these glob patterns, e.g. `["/docs/**"]`

**Returns:** `{ url, pages: [{ url, markdown }] }`

### batch

`olostepBatch()`. Scrape a list of URLs in one batch and return each page's markdown. The tool waits for the batch to finish before returning.

**Input:**

**urls** `array` *required*

The URLs to scrape (1 to 20)

**Returns:** `{ pages: [{ url, markdown }] }`

> [!NOTE]
>
> The crawl and batch tools are capped at 25 pages and 20 URLs per call to keep tool results small enough for the model. For larger jobs, call the [Crawls](https://docs.olostep.com/features/crawls.md) and [Batches](https://docs.olostep.com/features/batches.md) APIs directly with the [Node.js SDK](https://docs.olostep.com/sdks/node-js.md).

## Options

Every tool, and `olostepTools()`, accepts the same options:

```typescript theme={null}
olostepTools({
  apiKey: 'olostep_...',   // defaults to process.env.OLOSTEP_API_KEY
  maxCharacters: 20_000,   // max page content returned to the model, per page
  timeoutSeconds: 300,     // how long crawl and batch tools wait for the job to finish
});
```

**apiKey** `string`

Olostep API key. Defaults to the `OLOSTEP_API_KEY` environment variable.

**maxCharacters** `number` (default: 20000)

Maximum characters of page content returned to the model per page. Longer pages are truncated.

**timeoutSeconds** `number` (default: 300)

How long the crawl and batch tools wait for the job to finish, in seconds.

## Streaming Example

Use the same tools with `streamText`, for example in a Next.js route handler:

```typescript app/api/chat/route.ts theme={null}
import { convertToModelMessages, streamText, stepCountIs, type UIMessage } from 'ai';
import { olostepTools } from '@olostep/ai-sdk';

export async function POST(req: Request) {
  const { messages }: { messages: UIMessage[] } = await req.json();

  const result = streamText({
    model: 'openai/gpt-5-mini',
    system: 'You are a research assistant. Use the web tools and cite your sources.',
    messages: await convertToModelMessages(messages),
    tools: olostepTools(),
    stopWhen: stepCountIs(5),
  });

  return result.toUIMessageStreamResponse();
}
```

## Popular Use Cases

> **Research Assistant**
>
> 1. The model uses `search` or `answer` to find sources
> 2. It uses `scrape` to read the most relevant pages
> 3. It returns a summary with source URLs

> **Docs-Aware Coding Agent**
>
> 1. The model uses `map` with a `searchQuery` to find the right docs pages
> 2. It uses `batch` or `crawl` to read them
> 3. It writes code against the current API instead of guessing

> **Competitor and Price Monitoring**
>
> 1. A scheduled job calls `generateText` with `scrape` and `batch`
> 2. The model extracts prices or changes from each page
> 3. Your code compares them with the last run and sends alerts

## Best Practices

> **Only Pass the Tools You Need**
>
> Each tool's description is sent to the model. Fewer tools means fewer tokens and fewer wrong tool choices. For a chat that only reads pages, `scrape` and `search` are often enough.

> **Set a Step Limit**
>
> Always set `stopWhen` (for example `stepCountIs(5)`) so the model can chain tool calls but cannot loop forever.

> **Lower maxCharacters for Long Pages**
>
> Page content is the biggest part of the token bill. If you crawl or batch many pages, lower `maxCharacters` so all results fit in the model's context window.

> **Raise timeoutSeconds for Big Crawls**
>
> Crawl and batch tools wait for the job to finish. If you hit timeouts, raise `timeoutSeconds`, or lower `maxPages`.

## Troubleshooting

> **Authentication Failed**
>
> **Error**: "Invalid API key"
>
> **Solutions**:
>
> * Check your API key in the [dashboard](https://www.olostep.com/dashboard)
> * Make sure `OLOSTEP_API_KEY` is set where your code runs (for example in your Vercel project's environment variables)
> * Or pass the key directly with `olostepTools({ apiKey })`

> **The Model Doesn't Call the Tools**
>
> **Solutions**:
>
> * Set `stopWhen`, otherwise generation stops after the first step
> * Use a model that supports tool calling
> * Mention the web in your prompt or system message, for example "use the web tools and cite your sources"

> **Crawl or Batch Times Out**
>
> **Solutions**:
>
> * Raise `timeoutSeconds`
> * Lower `maxPages` or pass fewer URLs
> * On serverless functions, make sure the function's own time limit is longer than `timeoutSeconds`

> **Context Length Exceeded**
>
> **Solutions**:
>
> * Lower `maxCharacters`
> * Use `map` with `searchQuery` to pick fewer, more relevant pages before reading them

## Pricing

Olostep charges based on API usage, independent of the AI SDK:

* **Scrapes**: Pay per scrape
* **Searches and Answers**: Pay per request
* **Batches**: Pay per URL in batch
* **Crawls**: Pay per page crawled
* **Maps**: Pay per map operation

Check current pricing at [olostep.com/pricing](https://www.olostep.com/pricing).

## Support

> **[GitHub](https://github.com/olostep-api/olostep-ai-sdk)**
>
> Source code and issues

> **[Support Email](mailto:info@olostep.com)**
>
> Email: [info@olostep.com](mailto:info@olostep.com)

> **[AI SDK Docs](https://ai-sdk.dev/docs)**
>
> Learn about the Vercel AI SDK

> **[Documentation](https://docs.olostep.com)**
>
> Browse complete API docs

## Related Resources

> **[Scrapes API](https://docs.olostep.com/features/scrapes.md)**
>
> Learn about the Scrapes endpoint

> **[Crawls API](https://docs.olostep.com/features/crawls.md)**
>
> Learn about the Crawls endpoint

> **[Mastra Integration](https://docs.olostep.com/integrations/mastra.md)**
>
> Build agents with Mastra

> **[LangChain Integration](https://docs.olostep.com/integrations/langchain.md)**
>
> Build AI agents with LangChain


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.