使用 Olostep 的 Google 搜索抓取器
本指南演示如何使用 Olostep API 抓取 Google 搜索结果并将其解析为结构化的 JSON 数据。这对于自动化研究任务、收集竞争情报或构建需要搜索数据的应用程序特别有用。工作原理
下面的 Javascript 示例展示了如何使用 Google 搜索和 Olostep 的 google 搜索解析器@olostep/google-search 搜索特定人物（Patrick Collison）的 LinkedIn 个人资料 URL。
async function scrapeGoogleSearch(apiKey, query = "site%3Alinkedin.com+Patrick+Collison") {
const endpoint = "https://api.olostep.com/v1/scrapes";
const payload = {
"formats": ["json"],
"parser": {"id": "@olostep/google-search"},
"url_to_scrape": `https://www.google.com/search?q=${encodeURIComponent(query)}&gl=us&hl=en`,
"wait_before_scraping": 0,
};
const headers = {
"Authorization": `Bearer ${apiKey}`,
"Content-Type": "application/json"
};
try {
const response = await fetch(endpoint, {
method: "POST",
headers: headers,
body: JSON.stringify(payload)
});
const data = await response.json();
console.log(JSON.stringify(data, null, 4));
return data;
} catch (error) {
console.error("Error:", error);
throw error;
}
}
// 将 <API_KEY> 替换为你的实际 Olostep API 密钥
scrapeGoogleSearch("<API_KEY>");
响应格式
当你使用 Google 搜索解析器向 Olostep API 发出请求时，你将收到如下 JSON 响应：{
"id": "scrape_f2xghz17kt",
"object": "scrape",
"created": 1742679301,
"metadata": {},
"retrieve_id": "f2xghz17kt",
"url_to_scrape": "https://www.google.com/search?q=site%253Alinkedin.com%2BPatrick%2BCollison&gl=us&hl=en",
"result": {
"html_content": null,
"markdown_content": null,
"text_content": null,
"json_content": "{\"searchParameters\":{\"type\":\"search\",\"engine\":\"google\",\"q\":\"site:linkedin.com Patrick Collison\"},\"knowledgeGraph\":{\"description\":\"Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...\"},\"organic\":[{\"title\":\"Patrick Collison - Stripe\",\"link\":\"https://www.linkedin.com/in/patrickcollison\",\"position\":1,\"snippet\":\"Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...\",\"meta\":\"10.8K+ followers\"},{\"title\":\"The Stripe Story: How Patrick Collison Revolutionized ...\",\"link\":\"https://www.linkedin.com/pulse/stripe-story-how-patrick-collison-revolutionized-online-anshuman-jha-jzzic\",\"position\":2,\"snippet\":\"The Early Years: A Genius in the Making. Patrick Collison wasn't just bright—he was a supernova. By age 10, he'd devoured university-level math ...\"},{\"title\":\"In 2005, Patrick Collison was a 16-year-old winning ...\",\"link\":\"https://www.linkedin.com/posts/itselanagold_in-2005-patrick-collison-was-a-16-year-old-activity-7308533537576497154-w5vC\",\"position\":3,\"snippet\":\"In 2005, Patrick Collison was a 16-year-old winning Ireland's Young Scientist of the Year competition. By 2008, he and his younger brother ...\"},{\"title\":\"Patrick Collison on the importance of waiting a really long ...\",\"link\":\"https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-waiting-activity-7286001819145707520-1mdI\",\"position\":4,\"snippet\":\"Patrick argues you should also view every person you hire as bringing along another 50 people just like them if your company is successful.\"},{\"title\":\"Tim Ferriss' Post - Patrick Collison — CEO of Stripe (#353)\",\"link\":\"https://www.linkedin.com/posts/timferriss_patrick-collison-ceo-of-stripe-353-activity-7271892372358148096--dsK\",\"position\":5,\"snippet\":\"Author of 5 #1 NYT/WSJ bestsellers, early-stage investor, host of The Tim Ferriss Show podcast (1B+ downloads), and collector of the strange.\"},{\"title\":\"Patrick Collison wanted a guide to Stripe's culture that ...\",\"link\":\"https://www.linkedin.com/posts/first-round-capital_patrick-collison-wanted-a-guide-to-stripes-activity-7304833456948097024-Tt6h\",\"position\":6,\"snippet\":\"Patrick Collison wanted a guide to Stripe's culture that convinced 50% of candidates not to join. And Eeke de Milliano was tasked with ...\"},{\"title\":\"The Collison brothers (John & Patrick) explain why Stripe is ...\",\"link\":\"https://www.linkedin.com/posts/marcelvanoost_the-collison-brothers-john-patrick-explain-activity-7301586346349850624-L-4U\",\"position\":7,\"snippet\":\"The Collison brothers (John & Patrick) explain why Stripe is better off staying Private: \\\" This is our life's work. We're not going anywhere ...\"},{\"title\":\"Stripe CEO Patrick Collison on Crafting a Culture ...\",\"link\":\"https://www.linkedin.com/posts/jennifer-chatman-8086a918_stripe-ceo-patrick-collison-on-crafting-a-activity-7231753022849085440-0RE5\",\"position\":8,\"snippet\":\"When Patrick Collison and his brother John Collison founded digital payment company Stripe in 2010, he didn't come in with “any kind of ...\"},{\"title\":\"Patrick Collison on the importance of beauty and ...\",\"link\":\"https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-beauty-activity-7247935993817751552-Qt6h\",\"position\":9,\"snippet\":\"Patrick Collison on the importance of beauty and craftsmanship when building products “If Stripe is a monstrously successful business, ...\"}]}",
"llm_extract": null,
"screenshot_hosted_url": null,
"html_hosted_url": null,
"markdown_hosted_url": null,
"json_hosted_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json_f2xghz17kt.json",
"text_hosted_url": null,
"links_on_page": [],
"page_metadata": {
"status_code": 200,
"title": ""
}
}
}
- 基本请求信息：
id，object，created时间戳，url_to_scrape - 结果对象，包含访问不同数据格式的 URL
- json_content，包含结构化的搜索结果，包括：
searchParameters：关于搜索查询的信息knowledgeGraph：关于搜索主题的详细信息（如果可用）organic：搜索结果列表，包含标题、链接、位置和摘要peopleAlsoAsk：用户常搜索的相关问题relatedSearches：建议的相关搜索查询
json_content 是响应的主要部分，包含结构化的搜索结果。你可以直接从响应中访问 JSON 内容，或使用响应中提供的托管 URL。
结构化响应：json_content
{
"searchParameters": {
"type": "search",
"engine": "google",
"q": "site:linkedin.com Patrick Collison"
},
"knowledgeGraph": {
"description": "Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ..."
},
"organic": [
{
"title": "Patrick Collison - Stripe",
"link": "https://www.linkedin.com/in/patrickcollison",
"position": 1,
"snippet": "Experience. Stripe Graphic · Stripe. -. Education. Massachusetts Institute of Technology Graphic · Massachusetts Institute of Technology. 2006 - 2010 ...",
"meta": "10.8K+ followers"
},
{
"title": "The Stripe Story: How Patrick Collison Revolutionized ...",
"link": "https://www.linkedin.com/pulse/stripe-story-how-patrick-collison-revolutionized-online-anshuman-jha-jzzic",
"position": 2,
"snippet": "The Early Years: A Genius in the Making. Patrick Collison wasn't just bright—he was a supernova. By age 10, he'd devoured university-level math ..."
},
{
"title": "In 2005, Patrick Collison was a 16-year-old winning ...",
"link": "https://www.linkedin.com/posts/itselanagold_in-2005-patrick-collison-was-a-16-year-old-activity-7308533537576497154-w5vC",
"position": 3,
"snippet": "In 2005, Patrick Collison was a 16-year-old winning Ireland's Young Scientist of the Year competition. By 2008, he and his younger brother ..."
},
{
"title": "The Collison brothers (John & Patrick) explain why Stripe is ...",
"link": "https://www.linkedin.com/posts/marcelvanoost_the-collison-brothers-john-patrick-explain-activity-7301586346349850624-L-4U",
"position": 4,
"snippet": "The Collison brothers (John & Patrick) explain why Stripe is better off staying Private: \" This is our life's work. We're not going anywhere ..."
},
{
"title": "Patrick Collison on the importance of waiting a really long ...",
"link": "https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-waiting-activity-7286001819145707520-1mdI",
"position": 5,
"snippet": "Patrick argues you should also view every person you hire as bringing along another 50 people just like them if your company is successful."
},
{
"title": "Tim Ferriss' Post - Patrick Collison — CEO of Stripe (#353)",
"link": "https://www.linkedin.com/posts/timferriss_patrick-collison-ceo-of-stripe-353-activity-7271892372358148096--dsK",
"position": 6,
"snippet": "Author of 5 #1 NYT/WSJ bestsellers, early-stage investor, host of The Tim Ferriss Show podcast (1B+ downloads), and collector of the strange."
},
{
"title": "Patrick Collison on the importance of beauty and ...",
"link": "https://www.linkedin.com/posts/the-startup-archive_patrick-collison-on-the-importance-of-beauty-activity-7247935993817751552-Qt6h",
"position": 7,
"snippet": "Patrick Collison on the importance of beauty and craftsmanship when building products "If Stripe is a monstrously successful business, ..."
},
{
"title": "Stripe founder Patrick Collison tells the story of almost ...",
"link": "https://www.linkedin.com/posts/the-startup-archive_stripe-founder-patrick-collison-tells-the-activity-7235977194211000321-V-Cd",
"position": 8,
"snippet": "Stripe founder Patrick Collison tells the story of almost naming the company PayDemon Patrick and John Collison founded Stripe in 2010 when ..."
},
{
"title": "Patrick Collison created $50 billion of value as a co- ...",
"link": "https://www.linkedin.com/posts/tom-alder_patrick-collison-created-50-billion-of-value-activity-7239241304780513281-isxK",
"position": 9,
"snippet": "Patrick Collison created $50 billion of value as a co-founder of Stripe. He has also built the largest carbon removal program in the world."
}
]
}
json_hosted_url 字段访问 JSON 文件：
- 结构化 JSON：查看示例 JSON
formats 参数中包含这些格式，Olostep 将在响应中返回它们并提供每种格式的托管 URL。
示例使用场景
1. 查找 LinkedIn 个人资料
在上面的示例中，我们通过使用搜索查询site:linkedin.com Patrick Collison 来搜索 Patrick Collison 的 LinkedIn 个人资料。此技术可用于查找任何人的专业资料。
2. 研究公司
你可以修改查询以搜索公司信息：// 搜索公司信息
scrapeGoogleSearch(apiKey, "Stripe company information revenue");
3. 跟踪新闻文章
查找特定主题的最新新闻：// 搜索有关 AI 的最新新闻
scrapeGoogleSearch(apiKey, "artificial intelligence news after:2023-01-01");
4. 竞争分析
监控竞争对手的在线存在：// 搜索竞争对手提及
scrapeGoogleSearch(apiKey, "\"Company X\" vs \"Company Y\" comparison");
重要提示
- 搜索参数：
gl=us和hl=en参数将地理位置设置为美国，语言设置为英语。根据需要调整这些参数。
结论
一旦你拥有搜索结果数据，你可以：- 解析感兴趣的特定数据点
- 将结果存储在数据库中
- 构建自定义搜索界面
- 根据搜索结果触发操作
- 与其他 API 或服务集成