> ## Documentation Index
> Fetch the complete documentation index at: https://docs.olostep.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Olostep + Snowflake

> Search, scrape, map and crawl the live web from Snowflake SQL, and save pages as tables with the Olostep Native App.

The [Olostep Native App on the Snowflake Marketplace](https://app.snowflake.com/marketplace/listing/GZT8Z4URI87/olostep-olostep-web-search-scraping-and-crawling) brings live web data into Snowflake. [Olostep](https://www.olostep.com) is web data infrastructure for AI agents and enterprise AI workflows. Its APIs provide live web search, scraping and crawling to extract, clean and structure data from the web.

Call Olostep from SQL to search the web, scrape pages into clean markdown, map and crawl websites, and get answers with sources. Batch scrapes and crawls are saved as tables in your account, so you can join web data with your own data, load it into Cortex Search or use it in Cortex Agents.

[View on the Snowflake Marketplace →](https://app.snowflake.com/marketplace/listing/GZT8Z4URI87/olostep-olostep-web-search-scraping-and-crawling)

## Before you start

* **An Olostep account with an API key:** [get one free](https://www.olostep.com/dashboard), no credit card required. Your first 500 credits are included.
* **A Snowflake account with external access:** the app calls the Olostep API through an external access integration, which trial accounts don't support.
* **A role that can approve app connections:** approving the connection to `api.olostep.com` needs the `MANAGE APPLICATION SPECIFICATIONS` privilege, which `ACCOUNTADMIN` and `SECURITYADMIN` have.

## Setup

**1. Get the app**

Open the [listing](https://app.snowflake.com/marketplace/listing/GZT8Z4URI87/olostep-olostep-web-search-scraping-and-crawling) and click **Get**. The examples on this page assume you install it as `olostep`.

**2. Approve the connection to the Olostep API**

The app creates its own external access integration for `api.olostep.com`, and can't call the API until you approve it. In Snowsight, go to **Catalog** > **Apps** and select the app. Select the **Settings** icon, then open the **Configurations** tab. Under **External connections**, expand **Details** next to **External access integrations** and approve the `api.olostep.com` endpoint.

**3. Add your API key**

On the same **Configurations** tab, under **Credentials**, add a `GENERIC_STRING` secret that holds your API key from the [Olostep dashboard](https://www.olostep.com/dashboard/api-keys) to **Olostep API key**.

> [accordion] Prefer SQL? Do steps 2 and 3 in a worksheet
>
> ```sql theme={null}
> -- Approve the connection, using the sequence number that SHOW SPECIFICATIONS returns
> SHOW SPECIFICATIONS IN APPLICATION olostep;
> ALTER APPLICATION olostep APPROVE SPECIFICATION olostep_api_access SEQUENCE_NUMBER = 1;
>
> -- Store your key in a secret and give it to the app
> CREATE DATABASE IF NOT EXISTS olostep_config;
>
> CREATE OR REPLACE SECRET olostep_config.public.olostep_api_key
>   TYPE = GENERIC_STRING
>   SECRET_STRING = '<YOUR_OLOSTEP_API_KEY>';
>
> CALL olostep.core.register_reference('OLOSTEP_API_KEY', 'ADD',
>   SYSTEM$REFERENCE('SECRET', 'olostep_config.public.olostep_api_key', 'PERSISTENT', 'READ'));
> ```

## Procedures

| Procedure | What it does |
| - | - |
| `core.scrape(url)` | Scrape a page and return clean markdown (JavaScript rendering included) |
| `core.search(query, max_results)` | Search the live web and return titles, URLs and descriptions |
| `core.answer(task)` | Answer a question from the live web, with source URLs |
| `core.map(url, top_n)` | List the URLs on a website |
| `core.batch_scrape(urls, table_name)` | Scrape a list of URLs in one batch and save the pages to a table |
| `core.crawl(url, table_name, max_pages)` | Crawl a website and save the pages to a table |
| `core.save_results(id, table_name)` | Save the pages of a batch or crawl that was still running when the call returned |

Every procedure returns a `VARIANT`. On failure it returns an object with `status` set to `error`. Rate limits, server errors and timeouts are retried automatically.

## Usage

```sql theme={null}
-- Scrape a page into markdown
CALL olostep.core.scrape('https://docs.olostep.com');

-- Search the web
CALL olostep.core.search('latest AI agent frameworks', 5);

-- Answer a question with sources
CALL olostep.core.answer('What is the latest stable version of Next.js?');

-- List the URLs on a website
CALL olostep.core.map('https://docs.olostep.com', 50);
```

## Save pages as tables

`batch_scrape` and `crawl` save one row per page to `olostep.results.<table_name>`, with the columns `URL`, `MARKDOWN`, `ERROR` and `RETRIEVED_AT`. Using an existing table name replaces that table.

```sql theme={null}
-- Scrape every URL in one of your tables
CALL olostep.core.batch_scrape(
  (SELECT ARRAY_AGG(website) FROM my_db.crm.accounts),
  'account_pages');

-- Crawl a website (up to 25 pages)
CALL olostep.core.crawl('https://docs.olostep.com', 'olostep_docs', 25);

-- Query the results, or copy them into your own database
SELECT url, markdown FROM olostep.results.account_pages;
CREATE TABLE my_db.crm.account_pages AS SELECT * FROM olostep.results.account_pages;
```

Batches and crawls usually finish in a few minutes. If one is still running after 20 minutes, the procedure returns its `id` with `status` set to `in_progress`. Save its pages later with:

```sql theme={null}
CALL olostep.core.save_results('<id>', 'account_pages');
```

Tables in `results` are removed if you uninstall the app, so copy anything you want to keep into your own database.

## Troubleshooting

* **"Could not reach api.olostep.com":** the connection isn't approved yet. Approve it as in step 2.
* **"Unknown user-defined function INTERNAL.OLOSTEP\_REQUEST":** the API key isn't added yet. Add it as in step 3.
* **HTTP 401 in the error:** the secret doesn't hold a valid Olostep API key. Check it in the [Olostep dashboard](https://www.olostep.com/dashboard/api-keys).

## Resources

* [Olostep on the Snowflake Marketplace](https://app.snowflake.com/marketplace/listing/GZT8Z4URI87/olostep-olostep-web-search-scraping-and-crawling)
* [GitHub repository](https://github.com/olostep-api/olostep-snowflake)
* [Snowflake Native Apps documentation](https://docs.snowflake.com/en/developer-guide/native-apps/native-apps-about)
* [Olostep website](https://www.olostep.com)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.