使用 Olostep 的 Google 搜索抓取器

本指南演示如何使用 Olostep API 抓取 Google 搜索结果并将其解析为结构化的 JSON 数据。这对于自动化研究任务、收集竞争情报或构建需要搜索数据的应用程序特别有用。

工作原理

下面的 Javascript 示例展示了如何使用 Google 搜索和 Olostep 的 google search parser @olostep/google-search 搜索特定人物（Patrick Collison）的 LinkedIn 个人资料 URL。

async function scrapeGoogleSearch(apiKey, query = "site%3Alinkedin.com+Patrick+Collison") {
  const endpoint = "https://api.olostep.com/v1/scrapes";

  const payload = {
    "formats": ["json"],
    "parser": {"id": "@olostep/google-search"},
    "url_to_scrape": `https://www.google.com/search?q=${encodeURIComponent(query)}&gl=us&hl=en`,
    "wait_before_scraping": 0,
  };

  const headers = {
    "Authorization": `Bearer ${apiKey}`,
    "Content-Type": "application/json"
  };

  try {
    const response = await fetch(endpoint, {
      method: "POST",
      headers: headers,
      body: JSON.stringify(payload)
    });

    const data = await response.json();
    console.log(JSON.stringify(data, null, 4));
    return data;
  } catch (error) {
    console.error("Error:", error);
    throw error;
  }
}

// 将 <API_KEY> 替换为您实际的 Olostep API 密钥
scrapeGoogleSearch("<API_KEY>");

响应格式

当您使用 Google 搜索解析器向 Olostep API 发出请求时，您将收到如下面示例所示的 JSON 响应：

{
    "id": "scrape_f2xghz17kt",
    "object": "scrape",
    "created": 1742679301,
    "metadata": {},
    "retrieve_id": "f2xghz17kt",
    "url_to_scrape": "https://www.google.com/search?q=site%253Alinkedin.com%2BPatrick%2BCollison&gl=us&hl=en",
    "result": {
        "html_content": null,
        "markdown_content": null,
        "text_content": null,
        "json_content": "{\"searchParameters\":{\"type\":\"search\",\"engine\":\"google\",\"q\":\"site:linkedin.com Patrick Collison\"},\"knowledgeGraph\":{\"description\":\"Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...\"},\"organic\":[{\"title\":\"Patrick Collison - Stripe\",\"link\":\"https://www.linkedin.com/in/patrickcollison\",\"position\":1,\"snippet\":\"Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...\",\"meta\":\"10.8K+ followers\"},{\"title\":\"The Stripe Story: How Patrick Collison Revolutionized ...\",\"link\":\"https://www.linkedin.com/pulse/stripe-story-how-patrick-collison-revolutionized-online-anshuman-jha-jzzic\",\"position\":2,\"snippet\":\"The Early Years: A Genius in the Making. Patrick Collison wasn't just bright—he was a supernova. By age 10, he'd devoured university-level math ...\"},{\"title\":\"In 2005, Patrick Collison was a 16-year-old winning ...\",\"link\":\"https://www.linkedin.com/posts/itselanagold_in-2005-patrick-collison-was-a-16-year-old-activity-7308533537576497154-w5vC\",\"position\":3,\"snippet\":\"In 2005, Patrick Collison was a 16-year-old winning Ireland's Young Scientist of the Year competition. By 2008, he and his younger brother ...\"},{\"title\":\"Patrick Collison on the importance of waiting a really long ...\",\"link\":\"https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-waiting-activity-7286001819145707520-1mdI\",\"position\":4,\"snippet\":\"Patrick argues you should also view every person you hire as bringing along another 50 people just like them if your company is successful.\"},{\"title\":\"Tim Ferriss' Post - Patrick Collison — CEO of Stripe (#353)\",\"link\":\"https://www.linkedin.com/posts/timferriss_patrick-collison-ceo-of-stripe-353-activity-7271892372358148096--dsK\",\"position\":5,\"snippet\":\"Author of 5 #1 NYT/WSJ bestsellers, early-stage investor, host of The Tim Ferriss Show podcast (1B+ downloads), and collector of the strange.\"},{\"title\":\"Patrick Collison wanted a guide to Stripe's culture that ...\",\"link\":\"https://www.linkedin.com/posts/first-round-capital_patrick-collison-wanted-a-guide-to-stripes-activity-7304833456948097024-Tt6h\",\"position\":6,\"snippet\":\"Patrick Collison wanted a guide to Stripe's culture that convinced 50% of candidates not to join. And Eeke de Milliano was tasked with ...\"},{\"title\":\"The Collison brothers (John & Patrick) explain why Stripe is ...\",\"link\":\"https://www.linkedin.com/posts/marcelvanoost_the-collison-brothers-john-patrick-explain-activity-7301586346349850624-L-4U\",\"position\":7,\"snippet\":\"The Collison brothers (John & Patrick) explain why Stripe is better off staying Private: \\\" This is our life's work. We're not going anywhere ...\"},{\"title\":\"Stripe CEO Patrick Collison on Crafting a Culture ...\",\"link\":\"https://www.linkedin.com/posts/jennifer-chatman-8086a918_stripe-ceo-patrick-collison-on-crafting-a-activity-7231753022849085440-0RE5\",\"position\":8,\"snippet\":\"When Patrick Collison and his brother John Collison founded digital payment company Stripe in 2010, he didn't come in with “any kind of ...\"},{\"title\":\"Patrick Collison on the importance of beauty and ...\",\"link\":\"https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-beauty-activity-7247935993817751552-Qt6h\",\"position\":9,\"snippet\":\"Patrick Collison on the importance of beauty and craftsmanship when building products “If Stripe is a monstrously successful business, ...\"}]}",
        "llm_extract": null,
        "screenshot_hosted_url": null,
        "html_hosted_url": null,
        "markdown_hosted_url": null,
        "json_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json_f2xghz17kt.json",
        "text_hosted_url": null,
        "links_on_page": [],
        "page_metadata": {
            "status_code": 200,
            "title": ""
        }
    }
}

响应包含：

基本请求信息：id、object、created 时间戳、url_to_scrape
结果对象，包含访问不同格式数据的 URL
json_content，包含结构化的搜索结果，包括：
searchParameters：关于搜索查询的信息
knowledgeGraph：关于搜索对象的详细信息（如果有）
organic：搜索结果列表，包含标题、链接、位置和摘要
peopleAlsoAsk：用户常搜索的相关问题
relatedSearches：建议的相关搜索查询

json_content 是响应的主要部分，包含结构化的搜索结果。您可以直接从响应中访问 JSON 内容或使用响应中提供的托管 URL。

结构化响应：json_content

{
  "searchParameters": {
    "type": "search",
    "engine": "google",
    "q": "site:linkedin.com Patrick Collison"
  },
  "knowledgeGraph": {
    "description": "Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ..."
  },
  "organic": [
    {
      "title": "Patrick Collison - Stripe",
      "link": "https://www.linkedin.com/in/patrickcollison",
      "position": 1,
      "snippet": "Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...",
      "meta": "10.8K+ followers"
    },
    {
      "title": "The Stripe Story: How Patrick Collison Revolutionized ...",
      "link": "https://www.linkedin.com/pulse/stripe-story-how-patrick-collison-revolutionized-online-anshuman-jha-jzzic",
      "position": 2,
      "snippet": "The Early Years: A Genius in the Making. Patrick Collison wasn't just bright—he was a supernova. By age 10, he'd devoured university-level math ..."
    },
    {
      "title": "In 2005, Patrick Collison was a 16-year-old winning ...",
      "link": "https://www.linkedin.com/posts/itselanagold_in-2005-patrick-collison-was-a-16-year-old-activity-7308533537576497154-w5vC",
      "position": 3,
      "snippet": "In 2005, Patrick Collison was a 16-year-old winning Ireland's Young Scientist of the Year competition. By 2008, he and his younger brother ..."
    },
    {
      "title": "The Collison brothers (John & Patrick) explain why Stripe is ...",
      "link": "https://www.linkedin.com/posts/marcelvanoost_the-collison-brothers-john-patrick-explain-activity-7301586346349850624-L-4U",
      "position": 4,
      "snippet": "The Collison brothers (John & Patrick) explain why Stripe is better off staying Private: \" This is our life's work. We're not going anywhere ..."
    },
    {
      "title": "Patrick Collison on the importance of waiting a really long ...",
      "link": "https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-waiting-activity-7286001819145707520-1mdI",
      "position": 5,
      "snippet": "Patrick argues you should also view every person you hire as bringing along another 50 people just like them if your company is successful."
    },
    {
      "title": "Tim Ferriss' Post - Patrick Collison — CEO of Stripe (#353)",
      "link": "https://www.linkedin.com/posts/timferriss_patrick-collison-ceo-of-stripe-353-activity-7271892372358148096--dsK",
      "position": 6,
      "snippet": "Author of 5 #1 NYT/WSJ bestsellers, early-stage investor, host of The Tim Ferriss Show podcast (1B+ downloads), and collector of the strange."
    },
    {
      "title": "Patrick Collison on the importance of beauty and ...",
      "link": "https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-beauty-activity-7247935993817751552-Qt6h",
      "position": 7,
      "snippet": "Patrick Collison on the importance of beauty and craftsmanship when building products "If Stripe is a monstrously successful business, ..."
    },
    {
      "title": "Stripe founder Patrick Collison tells the story of almost ...",
      "link": "https://www.linkedin.com/posts/the-startup-archive_stripe-founder-patrick-collison-tells-the-activity-7235977194211000321-V-Cd",
      "position": 8,
      "snippet": "Stripe founder Patrick Collison tells the story of almost naming the company PayDemon Patrick and John Collison founded Stripe in 2010 when ..."
    },
    {
      "title": "Patrick Collison created $50 billion of value as a co- ...",
      "link": "https://www.linkedin.com/posts/tom-alder_patrick-collison-created-50-billion-of-value-activity-7239241304780513281-isxK",
      "position": 9,
      "snippet": "Patrick Collison created $50 billion of value as a co-founder of Stripe. He has also built the largest carbon removal program in the world."
    }
  ]
}

Olostep 还提供了一个托管的 JSON 文件，包含结构化的搜索结果。您可以使用响应中的 json_hosted_url 字段访问 JSON 文件：

结构化 JSON：查看示例 JSON

如果您还想获取搜索结果的 HTML 和 Markdown 内容，可以在 formats 参数中包含这些格式，Olostep 将在响应中返回它们并提供每种格式的托管 URL。

示例使用场景

1. 查找 LinkedIn 个人资料

在上面的示例中，我们通过使用搜索查询 site:linkedin.com Patrick Collison 来搜索 Patrick Collison 的 LinkedIn 个人资料。这种技术可以用于查找任何人的专业资料。

2. 研究公司

您可以修改查询以搜索公司信息：

// 搜索公司信息
scrapeGoogleSearch(apiKey, "Stripe company information revenue");

3. 跟踪新闻文章

查找关于特定主题的最新新闻：

// 搜索关于 AI 的最新新闻
scrapeGoogleSearch(apiKey, "artificial intelligence news after:2023-01-01");

4. 竞争分析

监控竞争对手的在线存在：

// 搜索竞争对手提及
scrapeGoogleSearch(apiKey, "\"Company X\" vs \"Company Y\" comparison");

重要提示

搜索参数：gl=us 和 hl=en 参数将地理位置设置为美国，语言设置为英语。根据需要调整这些参数。

结论

一旦您获得搜索结果数据，您可以：

解析感兴趣的特定数据点
将结果存储在数据库中
构建自定义搜索界面
根据搜索结果触发操作
与其他 API 或服务集成

如果您需要提取不同的数据点或自定义搜索行为，请通过 ‘info@olostep.com` 联系我们。

示例

​使用 Olostep 的 Google 搜索抓取器

​工作原理

​响应格式

​结构化响应：json_content

​示例使用场景

​1. 查找 LinkedIn 个人资料

​2. 研究公司

​3. 跟踪新闻文章

​4. 竞争分析

​重要提示

​结论