Skip to main content

​
Overview

Olostep’s scrape endpoint allows to extract content from any website. Content in markdown is useful if you want to feed it to an LLM without all the HTML. In this guide we will see how to extract markdown from a website like https://www.nea.com/team.

​
Prerequisites

Before getting started, ensure you have the following:
  • A valid Olostep API key. You can get one by signing up at Olostep.
  • Python installed on your system
  • The requests and json libraries (these come pre-installed with Python, but you can install them using pip install requests if needed)

​
Extracting Text from a Website

The following Python script demonstrates how to extract text and markdown content from a website using Olostep’s API. 
import requests
import json

url = "https://api.olostep.com/v1/scrapes"

payload = {
    "url_to_scrape": "https://www.nea.com/team",
    "country": "US",
    "formats": ["markdown"],
    "wait_before_scraping": 0,
    "remove_css_selectors": "default",
}

headers = {
    "Authorization": "Bearer <YOUR_API_KEY>",
    "Content-Type": "application/json"
}

response = requests.request("POST", url, json=payload, headers=headers)

print(json.dumps(response.json(), indent=4))

​
Example Response

A successful response will look something like this: 
{
    "id": "scrape_63x2e5sf5r",
    "object": "scrape",
    "created": 1740341743,
    "metadata": {},
    "retrieve_id": "63x2e5sf5r",
    "url_to_scrape": "https://www.nea.com/team",
    "result": {
        "html_content": null,
        "markdown_content": "NEA ….",
        "text_content": null,
        "json_content": null,
        "llm_extract": null,
        "screenshot_hosted_url": null,
        "html_hosted_url": null,
        "markdown_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/markDown_63x2e5sf5r.txt",
        "json_hosted_url": null,
        "text_hosted_url": null,
        "links_on_page": [],
        "page_metadata": {
            "status_code": 200,
            "title": ""
        }
    }
}

​
Explanation

  • url_to_scrape: specifies the website URL to extract content from.
  • formats: defines the output formats (text in this case).
  • Authorization: contains your API key to authenticate the request.
  • The response is formatted as JSON and printed for readability.

​
Conclusion

Using Olostep, you can easily extract markdown content from any website. This is useful if you want to get content from a website and feed it to an LLM for data extraction and analysis. If you want to extract content at scale from the same website over and over (e.g. monitoring data, price tracking, etc…) we recommend using a custom parser to get the content in JSON format.
⌘I