Scrapy-图像下载在第二次爬网中不起作用

Question

我写了一个蜘蛛，从网站下载数据，并点击链接获取详细数据。spider还使用默认的剪贴图像管道下载图像。到目前为止一切都很好。

但当我第二次（用另一个搜索词）启动蜘蛛时，图像下载就不再有效了。爬行工作正常。我没有任何错误。

这就是蜘蛛：

class DiscoSpider(BaseSpider):
    def __init__(self, query):
        super( BaseSpider, self ).__init__()
        self.name = "discogs"
        self.allowed_domains = ["discogs.com"]
        self.start_urls = [
            "http://www.discogs.com/search?q=%s&type=release" % query
        ]
    # parse all releases for the current search
    def parse(self, response):
        logging.debug('scrapy.parse')
        hxs = HtmlXPathSelector(response)
        li = hxs.select("//div[@id='page_content']/ol/li")
        items = []
        for l in li:
            item = DiscogsItem()
            ...
            # get the link for the callback for the tracklist
            link = l.select("a/@href").extract()[0]
            item['link'] = '' if link == None else link
            # get the img location
            img = l.select("a/img/@src").extract() 
            item['image_urls'] = [None] if img == None else img 
            # get the url for the tracklist callback
            url = urlparse.urljoin('%s%s' % ('http://www.', self.allowed_domains[0]), link)
            # request and callback to get tracklist for release
            item = Request(url, meta={'item':item}, callback=self.parse_tracklist)
            items.append(item)
            yield item        
    # callback to get the tracklist for each release
    def parse_tracklist(self, response):
        item = response.request.meta['item']
        hxs = HtmlXPathSelector(response)
        rows = hxs.select("//div[@class='section_content']/table[@class='playlist mini_playlist']/tr")
        tracklist = []
        for row in rows:
            track = {}
            title = row.select("td[@class='track']/span[@class='track_title']/text()").extract()
            track['title'] = '' if title in [None, '', []] else self.clean_track(title[0])
            ...
            tracklist.append(track)
        item['tracklist'] = tracklist    
        yield item

这是项目：

class DiscogsItem(Item):
    # define the fields for your item here like:
    link = Field()
    artist = Field()
    release = Field()
    label = Field()
    year = Field()
    tracklist = Field()
    image_urls = Field()
    images = Field()
    thumb = Field()

在我混乱的设置中：

ITEM_PIPELINES = ['scrapy.contrib.pipeline.images.ImagesPipeline']
IMAGES_STORE = '/home/f/work/py/discogs/tmp'
CONCURRENT_REQUESTS = 100
CONCURRENT_REQUESTS_PER_IP = 20
IMAGES_EXPIRES = 0

我在一个单独的过程中从PyQt UI运行spider，我是Scrapy/PyQt/StackOverflow的新手（很抱歉格式不正确）。

我在一个Xubuntu 12.04盒子上，里面有Python 2.7、PyQt4和Scrapy 0.12.0.2546。

有人知道为什么第二张图片下载不起作用吗？

提前谢谢。

Answer 1

我现在回答了我自己的问题，尽管我真的不知道问题是什么。我所做的是更改蜘蛛的构造函数，如下所示：

class DiscogsSpider(CrawlSpider):
    name = "discogs"
    allowed_domains = ["discogs.com"]
    def __init__(self, query):
        super( DiscogsSpider, self ).__init__()
        self.start_urls = [
            "http://www.discogs.com/search?q=%s&type=release" % query
        ]

我现在也扩展CrawlSpider而不是BasSpider。图像下载现在正在正常工作。

谢谢。