如何从网页正文中提取数据，其中我们在内容之间有动态谷歌广告使用刮擦

Question

我正在尝试从"拉利特卡拉学院奖学金2017 - 2018"中挑选身体的一部分......" 自 "电子邮件：[e-mail];[e-mail] 网站：lalitkala.gov.in">

但是我的输出是许多""和"\t"。我猜这是由于介于两者之间的广告词而发生的。知道如何解决这个问题吗？

import scrapy
class MySpider(scrapy.Spider):
    name = "test"
    start_urls = [
        'http://www.indiaeducation.net/scholarships/lalit-kala-akademi-scholarship.aspx',
    ]

    def parse(self, response):
        for scholarships in response.xpath('//*[@id="wrapper"]'):
            yield {
                  'text': scholarships.xpath('//*[@id="artBody"]/text()').extract(),
             }

Answer 1

像这样的东西？

>>> u" ".join(line.strip() for line in response.xpath('//div[@id="artBody"]//*[not(self::div)][not(self::script)]/text()').extract())
u'The Lalit Kala Akademi Scholarship 2017 - 2018 from the National Academy of Art, Delhi is awarded to learners who have passion for the visual arts. The scholarship is given by Lalit Kala Akademi, an Indian governmental institution for promotion and innovation of the visual arts. Presently 40 scholarships are offered by the National Academy of Art, however, this number may vary depending upon availability of resources.   Scholarship Through this scholarship the artists are given a work space to improve their skills and to develop new ideas within their field of visual art. Visual artists ( in disciplines such as graphic, sculpture, painting, ceramics), art historians and art critics may apply. The scholarship is worth Rs. 10,000 per month for a one-year period. Important dates Application available: available online now at lalitkala.gov.in  Last date for the filled application to reach the Akademi at New Delhi: 25 May, 2015  Eligibility criteria The Lalit Kala Akademi , New Delhi is looking for young rising artists in the visual arts, as well as art historians and art critics in the age group of 21-35 years Application procedure Prospective candidates should fill out the application form and include a Rs. 100 entry fee through postal order/ bank draft payable to Secretary, Lalit Kala Akademi drawn on New Delhi branch . A forum of senior artists decides who will be selected for the award. Selection is partly based on a published article or photographs of exhibits. The candidates awarded Lalit Kala Akademy Scholarship or National Academy of Art Scholarship need to work with regional centres of the Akademi listed below: Bhubaneswar  Guwahati  Kolkata  Lucknow  Delhi  Shimla  Contact details Lalit Kala Akademi Rabindra Bhavan, 35, Ferozeshah Road , New Delhi-110001 Telephone: 011 - 23009200 Fax : 011 - 23009292 Email: [e-mail]; [e-mail] Website: lalitkala.gov.in'