如何抓取按元数据中的关键字过滤的网站

Question

我写了一个抓取器，它应该只抓取关键字与给定网站匹配的网站。这是代码：

class MySpider(CrawlSpider):
    name = 'smm'
    allowed_domains = []
    start_urls = ['http://en.wikipedia.org/wiki/Social_media']
    rules = (
            Rule(SgmlLinkExtractor(deny=('statcounter.com/','wikipedia','play.google','books.google.com','github.com','amazon','bit.ly','wikimedia','mediawiki','creativecommons.org')), callback="parse_items", follow= True),
             )
    def parse_items(self, response):
        items = []
        #Define keywords present in metadata to scrape the webpage
        keywords = ['social media','social business','social networking','social marketing','online marketing','social selling',
            'social customer experience management','social cxm','social cem','social crm','google analytics','seo','sem',
            'digital marketing','social media manager','community manager']
        #Extract webpage keywords 
        metakeywords = response.xpath('//meta[@name="keywords"]').extract()
        #Discard empty keywords
        if metakeywords != []:
        #Compare keywords and extract if one of the defined keyboards is present in the metadata
            if (keywords in metaKW for metaKW in metakeywords):
                for link in response.xpath("//a"):
                    item = SocialMediaItem()
                    item['SourceTitle'] = link.xpath('/html/head/title').extract()
                    item['TargetTitle'] = link.xpath('text()').extract()
                    item['link'] = link.xpath('@href').extract()
                    item['webKW'] = metakeywords
                    outbound = str(link.xpath('@href').extract())
                    if 'http' in outbound:
                        items.append(item)
        return items

但是，我认为我错过了一些东西，因为它也会抓取没有 gicen 关键字的网站。你能帮忙解决这个问题吗？谢谢！

丹妮

Answer 1

如果要

检查metakeywords列表中是否有任何关键字，请使用任何关键字：

if any(key in metakeywords for key in keywords): 

Answer 2

我认为它在这个奇怪的if语句中。

if (keywords in metaKW for metaKW in metakeywords)

试试这个：

for metaKW in metakeywords:
    if metaKW in keywords:
       # code...
       break

这样就没有理由检查是否有元素。因此，您可以删除if metakeywords != []