我如何从这个维基百科表拉正确的图像URL

Question

我构建了一个scraper来从Wikipedia表中提取所有信息并将其上传到我的数据库。一切都很好，直到我意识到我在图像上拉错了URL，我想要实际的图像URL"http://upload.wikimedia.org/wikipedia/commons/thumb/3/38/Baconbutty.jpg"，而不是"/wiki/File: baconbutton .jpg"。下面是我到目前为止的代码:

def initialize
  @url = "http://en.wikipedia.org/wiki/List_of_sandwiches"
  @nodes = Nokogiri::HTML(open(@url))  
end
def summary
  sammich_data = @nodes
  sammiches = sammich_data.css('div.mw-content-ltr table.wikitable tr') 
    sammich_data.search('sup').remove
    sammich_hashes = sammiches.map {|x| 
      if content = x.css('td')[0]
        name = content.text
      end
      if content = x.css('td a.image').map {|link| link ['href']}
        image =content[0]
      end
      if content = x.css('td')[2]
        origin = content.text
      end
      if content = x.css('td')[3]
        description =content.text
      end

我的问题是这一行:

if content = x.css('td a.image').map {|link| link ['href']}
            image =content[0]

如果我去到td a.image img，它只给我一个null项

有什么建议吗?

Answer 1

我是这样做的(如果我要刮维基百科，我不会，因为他们有这个东西的API):

require 'nokogiri'
require 'open-uri'
require 'pp'
doc = Nokogiri::HTML(open("http://en.wikipedia.org/wiki/List_of_sandwiches"))  
sammich_hashes = doc.css('table.wikitable tr').map { |tr| 
  name, image, origin, description = tr.css('td,th')
  name, origin, description = [name, origin, description].map{ |n| n && n.text ? n.text : nil }
  image = image.at('img')['src'] rescue nil
  {
    name: name,
    origin: origin,
    description: description,
    image: image
  }
}
pp sammich_hashes

输出:

[
  {:name=>"Name", :origin=>"Origin", :description=>"Description", :image=>nil},
  {
    :name=>"Bacon",
    :origin=>"United Kingdom",
    :description=>"Often served with ketchup or brown sauce",
    :image=>"//upload.wikimedia.org/wikipedia/commons/thumb/3/38/Baconbutty.jpg/120px-Baconbutty.jpg"
  },
  ... [lots removed] ...
{
    :name=>"Zapiekanka",
    :origin=>"Poland",
    :description=>"A halved baguette or other bread usually topped with mushrooms and cheese, ham or other meats, and vegetables",
    :image=>"//upload.wikimedia.org/wikipedia/commons/thumb/1/12/Zapiekanka_3..jpg/120px-Zapiekanka_3..jpg"
  }
]

如果图像不可用，该字段将在返回的散列中设置为nil。

Answer 2

您可以使用img元素的srcset属性，拆分它并保留一个可用的调整大小的图像。

if content = x.at_css('td a.image img')
  image =content['srcset'].split(' 1.5x,').first