> ## Documentation Index
> Fetch the complete documentation index at: https://docs.olostep.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Maps

> Get all the URLs on a website.

Through the Olostep `/v1/maps` endpoint you can get all the URLs on a website. This is useful for content discovery, site structure analysis (e.g., SEO), or deciding which URLs you want to scrape next.

* Get all URLs on a website (including sitemaps and discovered links)
* Use special patterns to include/exclude paths (e.g. `/blog/**`)
* Paginate large responses with `cursor` (up to 10MB per response)
* Limit volume with `top_n`

For API details see the [Map Endpoint API Reference](/api-reference/maps/create).

## Installation

<CodeGroup>
  ```python Python theme={null}
  pip install olostep
  ```

  ```javascript Node theme={null}
  npm install olostep
  ```

  ```bash cURL theme={null}
  # curl is available by default on macOS, Linux, and Windows
  ```

  ```javascript Node (API) theme={null}
  npm install node-fetch
  ```

  ```bash Python (API) theme={null}
  pip install requests
  ```
</CodeGroup>

## Usage

Send a POST request with the website `url`. Optionally pass `include_urls`, `exclude_urls` (glob patterns), and `top_n`.

<CodeGroup>
  ```python Python theme={null}
  from olostep import Olostep

  client = Olostep(api_key="YOUR_REAL_KEY")

  sitemap = client.maps.create(url="https://docs.olostep.com")

  for url in sitemap.urls():
      print(url)
  ```

  ```js Node theme={null}
  import Olostep from 'olostep'

  const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })

  const map = await client.maps.create({ url: 'https://docs.olostep.com' })

  for await (const url of map.urls()) {
    console.log(url)
  }
  ```

  ```bash cURL theme={null}
  curl -s -X POST "https://api.olostep.com/v1/maps" \
    -H "Authorization: Bearer $OLOSTEP_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://docs.olostep.com"
    }'
  ```

  ```js Node (API) theme={null}
  const res = await fetch('https://api.olostep.com/v1/maps', {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer <YOUR_API_KEY>',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ url: 'https://docs.olostep.com' })
  })
  console.log(await res.json())
  ```

  ```python Python (API) theme={null}
  import requests
  import json

  endpoint = "https://api.olostep.com/v1/maps"

  payload = {
      "url": "https://docs.olostep.com"
  }
  headers = {
      "Authorization": "Bearer <YOUR_API_KEY>",
      "Content-Type": "application/json"
  }

  response = requests.post(endpoint, json=payload, headers=headers)
  print(json.dumps(response.json(), indent=2))
  ```
</CodeGroup>

The response time is typically within seconds but can take up to 120 seconds for more complex websites. It can extract all URLs from a website, even backlinks and those not present in the Sitemaps. You can also also decide the URLs paths you want to include or exclude from the response.

By default the endpoint returns around 100k URLs in a single call (10MB max). If the response includes more data, the API returns a `cursor` parameter which can be used for pagination and getting the subsequent URLs. For more details refer to the [API Reference](/api-reference/maps/create#body-cursor)

This endpoint is particularly useful when you need to:

* Discover all content pages on a website
* Analyze site structure and hierarchy
* Prepare URLs for batch processing
* Decide which specific URLs to scrape

For more fine-grained control over the URLs returned you can use the params `include_urls` and  `exclude_urls`.

### Example

Let's say that from [www.brex.com](http://www.brex.com) you want to extract all the urls that have the paths after `/product/` e.g `https://www.brex.com/product/api/no-code` but also include `www.brex.com/product`.
You can use the following code:

<CodeGroup>
  ```python Python theme={null}
  from olostep import Olostep

  client = Olostep(api_key="YOUR_REAL_KEY")

  sitemap = client.maps.create(
      url="https://www.brex.com/",
      include_urls=["/product", "/product/**"],
      top_n=100000,
  )

  for url in sitemap.urls():
      print(url)
  ```

  ```js Node theme={null}
  import Olostep from 'olostep'

  const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })

  const map = await client.maps.create({
    url: 'https://www.brex.com/',
    includeUrls: ['/product', '/product/**'],
    topN: 100000,
  })

  for await (const url of map.urls()) {
    console.log(url)
  }
  ```

  ```bash cURL theme={null}
  curl -s -X POST "https://api.olostep.com/v1/maps" \
    -H "Authorization: Bearer $OLOSTEP_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://www.brex.com/",
      "include_urls": ["/product", "/product/**"],
      "top_n": 100000
    }'
  ```

  ```js Node (API) theme={null}
  const endpoint = 'https://api.olostep.com/v1/maps'

  const res = await fetch(endpoint, {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer <YOUR_API_KEY>',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      url: 'https://www.brex.com/',
      include_urls: ['/product', '/product/**'],
      top_n: 100000
    })
  })
  console.log(await res.text())
  ```

  ```python Python (API) theme={null}
  import requests

  endpoint = "https://api.olostep.com/v1/maps"

  payload = {
      "url": "https://www.brex.com/",
      "include_urls": ["/product", "/product/**"],
      "top_n": 100000
  }

  headers = {
      "Authorization": "Bearer <YOUR_API_KEY>",
      "Content-Type": "application/json"
  }

  response = requests.post(endpoint, json=payload, headers=headers)

  print(response.text)
  ```
</CodeGroup>

## Conclusion

The maps endpoint is a powerful tool for content discovery and site analysis. It provides a comprehensive list of URLs on a website, enabling you to extract content from specific pages or analyze the site structure. This endpoint is particularly useful for SEO professionals, content marketers, AI agents who need to analyze website content or structure.

## Pricing

Map costs 1 credit. Then for every extra 1000 URLs returned in the response, an additional credit is billed.