Google Zoekresultaten Scraper met Olostep

Deze gids laat zien hoe je de Olostep API kunt gebruiken om Google zoekresultaten te scrapen en deze om te zetten in gestructureerde JSON-data. Dit is vooral nuttig voor het automatiseren van onderzoekstaken, het verzamelen van concurrentie-informatie, of het bouwen van applicaties die zoekdata vereisen.

Hoe Het Werkt

Het onderstaande voorbeeld in Javascript laat zien hoe je naar een LinkedIn-profiel URL van een specifieke persoon (Patrick Collison) kunt zoeken met behulp van Google zoekopdrachten en Olostep’s google zoekparser @olostep/google-search

async function scrapeGoogleSearch(apiKey, query = "site%3Alinkedin.com+Patrick+Collison") {
  const endpoint = "https://api.olostep.com/v1/scrapes";

  const payload = {
    "formats": ["json"],
    "parser": {"id": "@olostep/google-search"},
    "url_to_scrape": `https://www.google.com/search?q=${encodeURIComponent(query)}&gl=us&hl=en`,
    "wait_before_scraping": 0,
  };

  const headers = {
    "Authorization": `Bearer ${apiKey}`,
    "Content-Type": "application/json"
  };

  try {
    const response = await fetch(endpoint, {
      method: "POST",
      headers: headers,
      body: JSON.stringify(payload)
    });

    const data = await response.json();
    console.log(JSON.stringify(data, null, 4));
    return data;
  } catch (error) {
    console.error("Error:", error);
    throw error;
  }
}

// Vervang <API_KEY> met je eigen Olostep API-sleutel
scrapeGoogleSearch("<API_KEY>");

Responsformaat

Wanneer je een verzoek doet aan de Olostep API met de Google Zoekparser, ontvang je een JSON-respons zoals het onderstaande voorbeeld:

{
    "id": "scrape_f2xghz17kt",
    "object": "scrape",
    "created": 1742679301,
    "metadata": {},
    "retrieve_id": "f2xghz17kt",
    "url_to_scrape": "https://www.google.com/search?q=site%253Alinkedin.com%2BPatrick%2BCollison&gl=us&hl=en",
    "result": {
        "html_content": null,
        "markdown_content": null,
        "text_content": null,
        "json_content": "{\"searchParameters\":{\"type\":\"search\",\"engine\":\"google\",\"q\":\"site:linkedin.com Patrick Collison\"},\"knowledgeGraph\":{\"description\":\"Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...\"},\"organic\":[{\"title\":\"Patrick Collison - Stripe\",\"link\":\"https://www.linkedin.com/in/patrickcollison\",\"position\":1,\"snippet\":\"Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...\",\"meta\":\"10.8K+ followers\"},{\"title\":\"The Stripe Story: How Patrick Collison Revolutionized ...\",\"link\":\"https://www.linkedin.com/pulse/stripe-story-how-patrick-collison-revolutionized-online-anshuman-jha-jzzic\",\"position\":2,\"snippet\":\"The Early Years: A Genius in the Making. Patrick Collison wasn't just bright—he was a supernova. By age 10, he'd devoured university-level math ...\"},{\"title\":\"In 2005, Patrick Collison was a 16-year-old winning ...\",\"link\":\"https://www.linkedin.com/posts/itselanagold_in-2005-patrick-collison-was-a-16-year-old-activity-7308533537576497154-w5vC\",\"position\":3,\"snippet\":\"In 2005, Patrick Collison was a 16-year-old winning Ireland's Young Scientist of the Year competition. By 2008, he and his younger brother ...\"},{\"title\":\"Patrick Collison on the importance of waiting a really long ...\",\"link\":\"https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-waiting-activity-7286001819145707520-1mdI\",\"position\":4,\"snippet\":\"Patrick argues you should also view every person you hire as bringing along another 50 people just like them if your company is successful.\"},{\"title\":\"Tim Ferriss' Post - Patrick Collison — CEO of Stripe (#353)\",\"link\":\"https://www.linkedin.com/posts/timferriss_patrick-collison-ceo-of-stripe-353-activity-7271892372358148096--dsK\",\"position\":5,\"snippet\":\"Author of 5 #1 NYT/WSJ bestsellers, early-stage investor, host of The Tim Ferriss Show podcast (1B+ downloads), and collector of the strange.\"},{\"title\":\"Patrick Collison wanted a guide to Stripe's culture that ...\",\"link\":\"https://www.linkedin.com/posts/first-round-capital_patrick-collison-wanted-a-guide-to-stripes-activity-7304833456948097024-Tt6h\",\"position\":6,\"snippet\":\"Patrick Collison wanted a guide to Stripe's culture that convinced 50% of candidates not to join. And Eeke de Milliano was tasked with ...\"},{\"title\":\"The Collison brothers (John & Patrick) explain why Stripe is ...\",\"link\":\"https://www.linkedin.com/posts/marcelvanoost_the-collison-brothers-john-patrick-explain-activity-7301586346349850624-L-4U\",\"position\":7,\"snippet\":\"The Collison brothers (John & Patrick) explain why Stripe is better off staying Private: \\\" This is our life's work. We're not going anywhere ...\"},{\"title\":\"Stripe CEO Patrick Collison on Crafting a Culture ...\",\"link\":\"https://www.linkedin.com/posts/jennifer-chatman-8086a918_stripe-ceo-patrick-collison-on-crafting-a-activity-7231753022849085440-0RE5\",\"position\":8,\"snippet\":\"When Patrick Collison and his brother John Collison founded digital payment company Stripe in 2010, he didn't come in with “any kind of ...\"},{\"title\":\"Patrick Collison on the importance of beauty and ...\",\"link\":\"https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-beauty-activity-7247935993817751552-Qt6h\",\"position\":9,\"snippet\":\"Patrick Collison on the importance of beauty and craftsmanship when building products “If Stripe is a monstrously successful business, ...\"}]}",
        "llm_extract": null,
        "screenshot_hosted_url": null,
        "html_hosted_url": null,
        "markdown_hosted_url": null,
        "json_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json_f2xghz17kt.json",
        "text_hosted_url": null,
        "links_on_page": [],
        "page_metadata": {
            "status_code": 200,
            "title": ""
        }
    }
}

De respons bevat:

Basisverzoekinformatie: id, object, created timestamp, url_to_scrape
Resultaatobject met URL’s om toegang te krijgen tot verschillende formaten van de data
json_content met gestructureerde zoekresultaten inclusief:
searchParameters: Informatie over de zoekopdracht
knowledgeGraph: Gedetailleerde informatie over het zoekonderwerp (indien beschikbaar)
organic: Lijst van zoekresultaten met titel, link, positie en snippet
peopleAlsoAsk: Gerelateerde vragen die gebruikers vaak zoeken
relatedSearches: Voorgestelde gerelateerde zoekopdrachten

json_content is het belangrijkste deel van de respons met de gestructureerde zoekresultaten. Je kunt de JSON-content direct vanuit de respons openen of de gehoste URL gebruiken die in de respons wordt gegeven.

Gestructureerde Respons: json_content

{
  "searchParameters": {
    "type": "search",
    "engine": "google",
    "q": "site:linkedin.com Patrick Collison"
  },
  "knowledgeGraph": {
    "description": "Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ..."
  },
  "organic": [
    {
      "title": "Patrick Collison - Stripe",
      "link": "https://www.linkedin.com/in/patrickcollison",
      "position": 1,
      "snippet": "Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...",
      "meta": "10.8K+ followers"
    },
    {
      "title": "The Stripe Story: How Patrick Collison Revolutionized ...",
      "link": "https://www.linkedin.com/pulse/stripe-story-how-patrick-collison-revolutionized-online-anshuman-jha-jzzic",
      "position": 2,
      "snippet": "The Early Years: A Genius in the Making. Patrick Collison wasn't just bright—he was a supernova. By age 10, he'd devoured university-level math ..."
    },
    {
      "title": "In 2005, Patrick Collison was a 16-year-old winning ...",
      "link": "https://www.linkedin.com/posts/itselanagold_in-2005-patrick-collison-was-a-16-year-old-activity-7308533537576497154-w5vC",
      "position": 3,
      "snippet": "In 2005, Patrick Collison was a 16-year-old winning Ireland's Young Scientist of the Year competition. By 2008, he and his younger brother ..."
    },
    {
      "title": "The Collison brothers (John & Patrick) explain why Stripe is ...",
      "link": "https://www.linkedin.com/posts/marcelvanoost_the-collison-brothers-john-patrick-explain-activity-7301586346349850624-L-4U",
      "position": 4,
      "snippet": "The Collison brothers (John & Patrick) explain why Stripe is better off staying Private: \" This is our life's work. We're not going anywhere ..."
    },
    {
      "title": "Patrick Collison on the importance of waiting a really long ...",
      "link": "https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-waiting-activity-7286001819145707520-1mdI",
      "position": 5,
      "snippet": "Patrick argues you should also view every person you hire as bringing along another 50 people just like them if your company is successful."
    },
    {
      "title": "Tim Ferriss' Post - Patrick Collison — CEO of Stripe (#353)",
      "link": "https://www.linkedin.com/posts/timferriss_patrick-collison-ceo-of-stripe-353-activity-7271892372358148096--dsK",
      "position": 6,
      "snippet": "Author of 5 #1 NYT/WSJ bestsellers, early-stage investor, host of The Tim Ferriss Show podcast (1B+ downloads), and collector of the strange."
    },
    {
      "title": "Patrick Collison on the importance of beauty and ...",
      "link": "https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-beauty-activity-7247935993817751552-Qt6h",
      "position": 7,
      "snippet": "Patrick Collison on the importance of beauty and craftsmanship when building products "If Stripe is a monstrously successful business, ..."
    },
    {
      "title": "Stripe founder Patrick Collison tells the story of almost ...",
      "link": "https://www.linkedin.com/posts/the-startup-archive_stripe-founder-patrick-collison-tells-the-activity-7235977194211000321-V-Cd",
      "position": 8,
      "snippet": "Stripe founder Patrick Collison tells the story of almost naming the company PayDemon Patrick and John Collison founded Stripe in 2010 when ..."
    },
    {
      "title": "Patrick Collison created $50 billion of value as a co- ...",
      "link": "https://www.linkedin.com/posts/tom-alder_patrick-collison-created-50-billion-of-value-activity-7239241304780513281-isxK",
      "position": 9,
      "snippet": "Patrick Collison created $50 billion of value as a co-founder of Stripe. He has also built the largest carbon removal program in the world."
    }
  ]
}

Olostep biedt ook een gehost JSON-bestand met de gestructureerde zoekresultaten. Je kunt toegang krijgen tot het JSON-bestand met behulp van het json_hosted_url veld in de respons:

Gestructureerde JSON: Bekijk voorbeeld JSON

Als je ook de HTML- en Markdown-content van de zoekresultaten wilt verkrijgen, kun je deze formaten opnemen in de formats parameter en Olostep zal ze in de respons retourneren en gehoste URL’s voor elk formaat bieden.

Voorbeeldgebruiksscenario’s

1. LinkedIn-profielen Vinden

In het bovenstaande voorbeeld zoeken we naar Patrick Collison’s LinkedIn-profiel door gebruik te maken van de zoekopdracht site:linkedin.com Patrick Collison. Deze techniek kan worden gebruikt om professionele profielen voor elke persoon te vinden.

2. Onderzoek naar Bedrijven

Je kunt de zoekopdracht aanpassen om naar bedrijfsinformatie te zoeken:

// Zoek naar bedrijfsinformatie
scrapeGoogleSearch(apiKey, "Stripe company information revenue");

3. Nieuwsartikelen Volgen

Vind recent nieuws over een specifiek onderwerp:

// Zoek naar recent nieuws over AI
scrapeGoogleSearch(apiKey, "artificial intelligence news after:2023-01-01");

4. Concurrentieanalyse

Monitor de online aanwezigheid van concurrenten:

// Zoek naar vermeldingen van concurrenten
scrapeGoogleSearch(apiKey, "\"Company X\" vs \"Company Y\" comparison");

Belangrijke Opmerkingen

Zoekparameters: De gl=us en hl=en parameters stellen de geolocatie in op de VS en de taal op Engels. Pas deze aan indien nodig.

Conclusie

Zodra je de zoekresultaten hebt, kun je:

Specifieke datapunten van interesse parsen
De resultaten opslaan in een database
Een aangepaste zoekinterface bouwen
Acties triggeren op basis van zoekresultaten
Integreren met andere API’s of diensten

Als je verschillende datapunten moet extraheren of het zoekgedrag wilt aanpassen, neem dan contact op via ‘info@olostep.com`

Voorbeelden

Verkrijg Google Zoekresultaten (JS)

Google Zoekresultaten Scraper met Olostep

Hoe Het Werkt

Responsformaat

Gestructureerde Respons: json_content

Voorbeeldgebruiksscenario’s

1. LinkedIn-profielen Vinden

2. Onderzoek naar Bedrijven

3. Nieuwsartikelen Volgen

4. Concurrentieanalyse

Belangrijke Opmerkingen

Conclusie

​Google Zoekresultaten Scraper met Olostep

​Hoe Het Werkt

​Responsformaat

​Gestructureerde Respons: json_content

​Voorbeeldgebruiksscenario’s

​1. LinkedIn-profielen Vinden

​2. Onderzoek naar Bedrijven

​3. Nieuwsartikelen Volgen

​4. Concurrentieanalyse

​Belangrijke Opmerkingen

​Conclusie

Google Zoekresultaten Scraper met Olostep

Hoe Het Werkt

Responsformaat

Gestructureerde Respons: json_content

Voorbeeldgebruiksscenario’s

1. LinkedIn-profielen Vinden

2. Onderzoek naar Bedrijven

3. Nieuwsartikelen Volgen

4. Concurrentieanalyse

Belangrijke Opmerkingen

Conclusie