如何使用 Nokogiri 解析 Google 搜索结果

Question

我需要帮助从Google搜索结果中提取URL，并被告知使用Nokogiri。我安装了它并阅读了Nokogiri文档，但不知道从哪里开始 - 这对我来说都是希腊语。

我知道我正在寻找的是每个结果的URL，每个结果都存在于一个<cite>标签之间。到目前为止，我所能弄清楚的就是提取搜索结果，但我只是不知道如何从文件中提取特定数据。这是我确实拥有的微小代码：

serp = Nokogiri::HTML(open("http://www.google.com/search?num=100&q=stackoverflow"))

Answer 1

享受:)

require 'open-uri'
require 'nokogiri'
page = open "http://www.google.com/search?num=100&q=stackoverflow"
html = Nokogiri::HTML page
html.search("cite").each do |cite|
  puts cite.inner_text
end

另请查看野古吉里教程

Answer 2

确保你使用的是user-agent（标头），否则它将返回一个空输出，因为谷歌最终会阻止请求。什么是我的用户代理。

headers = {
  "User-Agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

在线 IDE 中的代码和示例：

require 'nokogiri'
require 'httparty'
require 'json'
headers = {
  "User-Agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
params = {
  q: "stackoverflow",
  num: "100"
}
response = HTTParty.get("https://www.google.com/search",
                        query: params,
                        headers: headers)
doc = Nokogiri::HTML(response.body)

data = doc.css(".tF2Cxc").map do |result|
  title = result.at_css(".DKV0Md")&.text
  link = result.at_css(".yuRUbf a")&.attr("href")
  displayed_link = result.at_css(".tjvcx")&.text
  snippet = result.at_css(".VwiC3b")&.text
  # puts "#{title}#{snippet}#{link}#{displayed_link}nn"
  {
    title: title,
    link: link,
    displayed_link: displayed_link,
    snippet: snippet,
  }.compact
end
puts JSON.pretty_generate(data)
--------
=begin
[
  {
    "title": "Stack for Stack Overflow - Apps on Google Play",
    "link": "https://play.google.com/store/apps/details?id=me.tylerbwong.stack&hl=en_US&gl=US",
    "displayed_link": "https://play.google.com › store › apps › details",
    "snippet": "Stack is powered by Stack Overflow and other Stack Exchange sites. Search and filter through questions to find the exact answer you're looking for!"
  }
...
]
=end

<小时 />

或者，你可以从SerpApi获得谷歌有机结果API。这是一个带有免费计划的付费 API。

主要区别在于无需弄清楚如何抓取页面的某些部分。所有需要做的只是迭代结构化的JSON字符串。

require 'google_search_results' 
require 'json'
params = {
  api_key: ENV["API_KEY"],
  engine: "google",
  q: "stackoverflow",
  hl: "en",
  num: "100"
}
search = GoogleSearch.new(params)
hash_results = search.get_hash
data = hash_results[:organic_results].map do |result|
  title = result[:title]
  link = result[:link]
  displayed_link = result[:displayed_link]
  snippet = result[:snippet]
  {
    title: title,
    link: link,
    displayed_link: displayed_link,
    snippet: snippet
  }.compact
end
  puts JSON.pretty_generate(data)

-------------
=begin
[
  {
    "title": "Stack Overflow - Home | Facebook",
    "link": "https://www.facebook.com/officialstackoverflow/",
    "displayed_link": "https://www.facebook.com › Pages › Interest",
    "snippet": "Stack Overflow. 519455 likes · 587 talking about this. We are the world's programmer community."
  }
...
]
=end

免责声明，我为 SerpApi 工作。