我写了以下蜘蛛来抓取webmd网站以供患者评论
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
class MySpider(BaseSpider):
name = "webmd"
allowed_domains = ["webmd.com"]
start_urls = ["http://www.webmd.com/drugs/drugreview-92884-Boniva"]
def parse(self, response):
hxs = HtmlXPathSelector(response)
titles = hxs.select("//p")
title = titles.select("//p[contains(@class, 'comment')and contains(@style, 'display:none')]/text()").extract()
print(title)
执行此代码为我提供了所需的输出,但有很多重复,即相同的注释至少重复 10 次。帮我解决这个问题。
你可以像这样重写你的爬虫代码:
import scrapy
# Your Items
class ReviewItem(scrapy.Item):
review = scrapy.Field()
class WebmdSpider(scrapy.Spider):
name = "webmd"
allowed_domains = ["webmd.com"]
start_urls = ['http://www.webmd.com/drugs/drugreview-92884-Boniva']
def parse(self, response):
titles = response.xpath('//p[contains(@id, "Full")]')
for title in titles:
item = ReviewItem()
item['review'] = title.xpath('text()').extract_first()
yield item
# Checks if there is a next page link, and keeping parsing if True
next_page = response.xpath('(//a[contains(., "Next")])[1]/@href').extract_first()
if next_page:
yield scrapy.Request(response.urljoin(next_page), callback=self.parse)
它只选择完整的客户评论,没有重复,并将它们保存在Scrapy Items中。注意:您可以使用更方便的快捷方式response
代替HtmlXPathSelector
。另外,我将已弃用的scrapy.BaseSpider
更改为scrapy.Spider
。
要将评论保存为 csv 格式,您只需使用 Scrapy Feed 导出并输入控制台scrapy crawl webmd -o reviews.csv
.
sets
来获取唯一的评论。我希望你知道选择器将结果作为list
返回,所以如果你使用集合,那么你只会得到唯一的结果。所以
def parse(self,response):
hxs = HtmlXPathSelector(response)
titles = hxs.select("//p")
title = set(titles.select("//p[contains(@class, 'comment')and contains(@style, 'display:none')]/text()").extract())
print (title) #this will have only unique results.