Get all the URLs on a website. Useful for site structure analysis or decide which url to scrape.
The maps endpoint allows you to get all URLs on a certain website. This is useful for content discovery, site structure analysis (e.g. for SEO) or decide which of those URLs you want to get the content from.
The response time is typically within seconds but can take up to 120 seconds for more complex websites. It can extract all URLs from a website, even backlinks and those not present in the Sitemaps. You can also also decide the URLs paths you want to include or exclude from the response.
By default the endpoint returns around 100k URLs in a single call (10MB max). If the response includes more data, the API returns a cursor
parameter which can be used for pagination and getting the subsequent urls. For more details refer to the API Reference
To extract URLs from a website, send a POST request with the url of the website to be processed. The endpoint typically responds within seconds, providing a comprehensive list of URLs.
The response will include all discovered URLs for the specified website. This endpoint is particularly useful when you need to:
For more fine-grained control over the URLs returned you can use the params include_urls
and exclude_urls
.
Let’s say that from www.brex.com you want to extract all the urls that have the paths after /product/
e.g https://www.brex.com/product/api/no-code
but also include www.brex.com/product
.
You can use the following code:
For more details refer to the API Reference
Each request sent to the maps endpoint is billed as 1 credit. Then for every extra 1000 urls returned in the response an additional credit is used.
The maps endpoint is a powerful tool for content discovery and site analysis. It provides a comprehensive list of URLs on a website, enabling you to extract content from specific pages or analyze the site structure. This endpoint is particularly useful for SEO professionals, content marketers, AI agents who need to analyze website content or structure.
Get all the URLs on a website. Useful for site structure analysis or decide which url to scrape.
The maps endpoint allows you to get all URLs on a certain website. This is useful for content discovery, site structure analysis (e.g. for SEO) or decide which of those URLs you want to get the content from.
The response time is typically within seconds but can take up to 120 seconds for more complex websites. It can extract all URLs from a website, even backlinks and those not present in the Sitemaps. You can also also decide the URLs paths you want to include or exclude from the response.
By default the endpoint returns around 100k URLs in a single call (10MB max). If the response includes more data, the API returns a cursor
parameter which can be used for pagination and getting the subsequent urls. For more details refer to the API Reference
To extract URLs from a website, send a POST request with the url of the website to be processed. The endpoint typically responds within seconds, providing a comprehensive list of URLs.
The response will include all discovered URLs for the specified website. This endpoint is particularly useful when you need to:
For more fine-grained control over the URLs returned you can use the params include_urls
and exclude_urls
.
Let’s say that from www.brex.com you want to extract all the urls that have the paths after /product/
e.g https://www.brex.com/product/api/no-code
but also include www.brex.com/product
.
You can use the following code:
For more details refer to the API Reference
Each request sent to the maps endpoint is billed as 1 credit. Then for every extra 1000 urls returned in the response an additional credit is used.
The maps endpoint is a powerful tool for content discovery and site analysis. It provides a comprehensive list of URLs on a website, enabling you to extract content from specific pages or analyze the site structure. This endpoint is particularly useful for SEO professionals, content marketers, AI agents who need to analyze website content or structure.