Olostep + Vercel AI SDK Integration

The Olostep AI SDK integration gives agents built with the Vercel AI SDK ready-made tools for live web data. Olostep is a web search, scraping and crawling API: an API to search, extract and structure web data. Pass the tools to generateText or streamText and the model decides when to search, scrape or crawl. Install from npm →

Features

The package provides 6 tools:

Scrape

Scrape any URL and return clean markdown, with JavaScript rendering

Search

Search the live web and return titles, URLs and descriptions

Answer

Answer a question from the live web, with source URLs

Map

List the URLs on a website, optionally ranked by a query

Crawl

Crawl a site from a starting URL and return the markdown of each page

Batch

Scrape a list of URLs in one batch and return the markdown of each

Installation

npm install @olostep/ai-sdk ai zod

yarn add @olostep/ai-sdk ai zod

pnpm add @olostep/ai-sdk ai zod

Works with AI SDK 5, 6 and 7, Zod 3.25+ or 4, and Node.js 18+.

Setup

Set your Olostep API key as an environment variable:

export OLOSTEP_API_KEY=your-api-key-here

Or in your .env file:

OLOSTEP_API_KEY=your-api-key-here

Get your API key from the Olostep Dashboard.

Quick Start

Pass all Olostep tools to the model with olostepTools():

import { generateText, stepCountIs } from 'ai';
import { olostepTools } from '@olostep/ai-sdk';

const { text } = await generateText({
  model: 'openai/gpt-5-mini',
  tools: olostepTools(),
  stopWhen: stepCountIs(5),
  prompt: 'Read the Next.js 15 release notes and list the breaking changes, with sources.',
});

console.log(text);

stopWhen lets the model call tools over several steps, for example search first and then scrape the best result. The model string uses the Vercel AI Gateway. You can also pass any AI SDK provider model, such as openai('gpt-5-mini') from @ai-sdk/openai.

Pick Individual Tools

Only give the model the tools it needs:

import { olostepScrape, olostepSearch } from '@olostep/ai-sdk';

const tools = {
  scrape: olostepScrape(),
  search: olostepSearch(),
};

Available Tools

The tool names below are the keys used by olostepTools().

scrape

olostepScrape(). Scrape a single URL and return its content as markdown. Input:

string

required

The URL of the page to scrape

number

Milliseconds to wait for JavaScript-heavy pages to render (0 to 10000)

Returns: { url, markdown }

search

olostepSearch(). Search the live web. Input:

string

required

The search query

number

default:"10"

Maximum number of results (1 to 20)

Returns: { query, results: [{ title, url, description }] }

answer

olostepAnswer(). Search and read the live web to answer a question, with sources. Input:

string

required

The question or research task

Returns: { answer, sources }

map

olostepMap(). Discover the URLs on a website. Input:

string

required

The website to map

string

Rank URLs by relevance to this query

number

Maximum number of URLs to return (1 to 500)

Returns: { url, urls }

crawl

olostepCrawl(). Crawl a website from a starting URL and return each page’s markdown. The tool waits for the crawl to finish before returning. Input:

string

required

The URL to start crawling from

number

default:"5"

Maximum pages to crawl (1 to 25)

array

Only crawl URLs matching these glob patterns, e.g. ["/docs/**"]

Returns: { url, pages: [{ url, markdown }] }

batch

olostepBatch(). Scrape a list of URLs in one batch and return each page’s markdown. The tool waits for the batch to finish before returning. Input:

array

required

The URLs to scrape (1 to 20)

Returns: { pages: [{ url, markdown }] }

The crawl and batch tools are capped at 25 pages and 20 URLs per call to keep tool results small enough for the model. For larger jobs, call the Crawls and Batches APIs directly with the Node.js SDK.

Options

Every tool, and olostepTools(), accepts the same options:

olostepTools({
  apiKey: 'olostep_...',   // defaults to process.env.OLOSTEP_API_KEY
  maxCharacters: 20_000,   // max page content returned to the model, per page
  timeoutSeconds: 300,     // how long crawl and batch tools wait for the job to finish
});

string

Olostep API key. Defaults to the OLOSTEP_API_KEY environment variable.

number

default:"20000"

Maximum characters of page content returned to the model per page. Longer pages are truncated.

number

default:"300"

How long the crawl and batch tools wait for the job to finish, in seconds.

Streaming Example

Use the same tools with streamText, for example in a Next.js route handler:

app/api/chat/route.ts

import { convertToModelMessages, streamText, stepCountIs, type UIMessage } from 'ai';
import { olostepTools } from '@olostep/ai-sdk';

export async function POST(req: Request) {
  const { messages }: { messages: UIMessage[] } = await req.json();

  const result = streamText({
    model: 'openai/gpt-5-mini',
    system: 'You are a research assistant. Use the web tools and cite your sources.',
    messages: await convertToModelMessages(messages),
    tools: olostepTools(),
    stopWhen: stepCountIs(5),
  });

  return result.toUIMessageStreamResponse();
}

Popular Use Cases

Research Assistant

The model uses search or answer to find sources
It uses scrape to read the most relevant pages
It returns a summary with source URLs

Docs-Aware Coding Agent

The model uses map with a searchQuery to find the right docs pages
It uses batch or crawl to read them
It writes code against the current API instead of guessing

Competitor and Price Monitoring

A scheduled job calls generateText with scrape and batch
The model extracts prices or changes from each page
Your code compares them with the last run and sends alerts

Best Practices

Only Pass the Tools You Need

Each tool’s description is sent to the model. Fewer tools means fewer tokens and fewer wrong tool choices. For a chat that only reads pages, scrape and search are often enough.

Set a Step Limit

Always set stopWhen (for example stepCountIs(5)) so the model can chain tool calls but cannot loop forever.

Lower maxCharacters for Long Pages

Page content is the biggest part of the token bill. If you crawl or batch many pages, lower maxCharacters so all results fit in the model’s context window.

Raise timeoutSeconds for Big Crawls

Crawl and batch tools wait for the job to finish. If you hit timeouts, raise timeoutSeconds, or lower maxPages.

Troubleshooting

Authentication Failed

Error: “Invalid API key”Solutions:

Check your API key in the dashboard
Make sure OLOSTEP_API_KEY is set where your code runs (for example in your Vercel project’s environment variables)
Or pass the key directly with olostepTools({ apiKey })

The Model Doesn't Call the Tools

Solutions:

Set stopWhen, otherwise generation stops after the first step
Use a model that supports tool calling
Mention the web in your prompt or system message, for example “use the web tools and cite your sources”

Crawl or Batch Times Out

Solutions:

Raise timeoutSeconds
Lower maxPages or pass fewer URLs
On serverless functions, make sure the function’s own time limit is longer than timeoutSeconds

Context Length Exceeded

Solutions:

Lower maxCharacters
Use map with searchQuery to pick fewer, more relevant pages before reading them

Pricing

Olostep charges based on API usage, independent of the AI SDK:

Scrapes: Pay per scrape
Searches and Answers: Pay per request
Batches: Pay per URL in batch
Crawls: Pay per page crawled
Maps: Pay per map operation

Check current pricing at olostep.com/pricing.

Support

GitHub

Source code and issues

Support Email

Email: info@olostep.com

AI SDK Docs

Learn about the Vercel AI SDK

Documentation

Browse complete API docs

Scrapes API

Learn about the Scrapes endpoint

Crawls API

Learn about the Crawls endpoint

Mastra Integration

Build agents with Mastra

LangChain Integration

Build AI agents with LangChain

​Features

Scrape

Search

Answer

Map

Crawl

Batch

​Installation

​Setup

​Quick Start

​Pick Individual Tools

​Available Tools

​scrape

​search

​answer

​map

​crawl

​batch

​Options

​Streaming Example

​Popular Use Cases

​Best Practices

​Troubleshooting

​Pricing

​Support

GitHub

Support Email

AI SDK Docs

Documentation

​Related Resources

Scrapes API

Crawls API

Mastra Integration

LangChain Integration

Features

Installation

Setup

Quick Start

Pick Individual Tools

Available Tools

scrape

search

answer

map

crawl

batch

Options

Streaming Example

Popular Use Cases

Best Practices

Troubleshooting

Pricing

Support

Related Resources