当有多个'p'标签时如何抓取？

Question

我试图刮一个网站，有多个<p>标签，将始终以单词"位于:…"开始。没有其他<p>标签以这些词开头。

我如何让我的刮板只提取那些特定的标签?

这是scraper.rb:

 require 'open-uri'
    require 'nokogiri'
    require 'csv'
    # Store URL to be scraped
    url = "http://www.timeout.com/london/restaurants/the-50-best-street-food-stalls-in-london?package_page=68111"
    # Parse the page with Nokogiri
    page = Nokogiri::HTML(open(url))
    # Display output onto the screen
    name =[]
    page.css('h3').each do |line|
      name << h3.text.strip
    end
    zero =[]
    page.css('p').each do |line|
      zero << line.text.strip
    end

这是要抓取的传入HTML:

      <div class="feature-item__text">
      <h3>
     Yu Kyu
      </h3>
                                                                                                                                                                                                                                    <p class="feature_item__annotation--truncated">
    <p>Everybody knows that on any given visit to...</p>
    <p><strong>Don't miss:</strong> Curry Katsu Sandwich (&pound;6.50).</p>
    <p><strong>Find them at:</strong><a href="http://www.timeout.com/london/restaurants/kerb">Kerb</a>.</p>
    <p><strong>But first check:</strong> <a href="...">@_YuKyu_</a></p>
</p>
                                                                                                            </div>
    </div>
    <div class="listing_meta_controls"></div>    
    </article>

Answer 1

你的问题中有几个问题，以及它如何与HTML对齐。

有可能该网站正在改变措辞以摆脱抓取者，并将"位于:"更改为"查找他们"。如果是这样，那么在定位所需信息时，您可能无法将其作为路标。

也就是说，CSS不允许我们查找以某些东西开头的文本，但是XPath可以:

@doc.search('//strong[starts-with(text(), "Find")]/following-sibling::a')

该选择器将定位所有<strong>Find them at:</strong>标签和相邻的兄弟<a>标签，允许您处理标签的text或'href'参数，具体取决于您想要的。使用这个选择器，我在页面上看到84个点击，看起来像:

@doc.search('//strong[starts-with(text(), "Find")]/following-sibling::a').first.to_html 
#=> "<a href="http://www.timeout.com/london/restaurants/kerb">Kerb</a>"
@doc.search('//strong[starts-with(text(), "Find")]/following-sibling::a').first.text 
#=> "Kerb"
@doc.search('//strong[starts-with(text(), "Find")]/following-sibling::a').first['href'] 
#=> "http://www.timeout.com/london/restaurants/kerb"

如果你想使用CSS，这是可能的，但你必须采取不同的策略。查找包含<div>的，然后在里面搜索:

require 'nokogiri'
require 'open-uri'
URL = 'http://www.timeout.com/london/restaurants/the-50-best-street-food-stalls-in-london?package_page=68111'
doc = Nokogiri::HTML(open(URL))
feature_items = doc.search('div.feature-item__text').map{ |div|
  h3 = div.at('h3').text.strip
  a = div.at('strong + a')
  a_text = a.text.strip
  a_href = a['href']
  {
    h3: h3,
    a_text: a_text,
    a_href: a_href
  }
}

返回一个哈希数组，每个哈希值代表一个特定位置的信息。

以下是发现的前五个:

feature_items[0, 5]
# => [{:h3=>"Yu Kyu",
#      :a_text=>"Kerb",
#      :a_href=>"http://www.timeout.com/london/restaurants/kerb"},
#     {:h3=>"Luardos",
#      :a_text=>"Kerb",
#      :a_href=>"http://www.timeout.com/london/restaurants/kerb"},
#     {:h3=>"Mission Mariscos",
#      :a_text=>"The Schoolyard",
#      :a_href=>"http://www.timeout.com/london/shopping/broadway-market-1"},
#     {:h3=>"Butchies",
#      :a_text=>"Broadway Market",
#      :a_href=>"http://www.timeout.com/london/shopping/broadway-market-1"},
#     {:h3=>"BBQ Dreamz",
#      :a_text=>"Kerb",
#      :a_href=>"http://www.timeout.com/london/restaurants/kerb"}]

Answer 2

如果我没理解错的话，你可以直接做

zero =[]
page.css('p').each do |line|
   text = line.text.strip
   if text.present? && text.include? 'Located in'
     zero << text
   end
end