Create Scrape
Scrape a url with provided configuration and get content.
Authorizations
Bearer authentication header of the form Bearer <token>
, where <token>
is your auth token.
Body
The URL to start scraping from.
Time to wait in milliseconds before starting the scraping.
Formats in which you want the content.
html
, markdown
, text
, parser_extract
, raw_pdf
Option to remove certain CSS selectors from the content. Optionally, you can also pass a JSON stringified array of specific selectors you want to remove. The CSS selectors removed when this option is set to default are ['nav','footer','script','style','noscript','svg',[role=alert],[role=banner],[role=dialog],[role=alertdialog],[role=region][aria-label*=skip i],[aria-modal=true]]
default
, none
, array
Actions to perform on the page before getting the content.
Residential country to load the request from.
Supported values are:
- US (United States)
- CA (Canada)
- IT (Italy)
- IN (India)
- GB (England)
- JP (Japan)
- MX (Mexico)
- AU (Australia)
- ID (Indonesia)
- UA (UAE)
- RU (Russia)
- RANDOM
Some operations, like scraping Google Search and Google News, support all countries.
Specify the HTML transformer to use, if any. Postlight's Mercury Parser library is used to remove ads and other unwanted content from the scraped content.
postlight
, none
Option to remove images from the scraped content. Defaults to false.
List of class names to remove from the content.
Configuration for parser extraction.
With this option, you can get all the links present on the page you scrape.
Configuration for screen size. Preset dimensions are available through screen_type: desktop (1920x1080), mobile (414x896), or default (768x1024).
User-defined metadata. Not supported yet