收益要求呼叫产生的递归方法的怪异结果

Question

我正在尝试使用Python和scrapy在所有国家的所有机场中刮掉所有出发和到达。

当一个机场出发或到达> 100时，此著名站点(飞行雷达(使用的JSON数据库需要查询页面。我还根据查询的实际日utc计算时间戳。

我尝试使用此层次结构创建一个数据库：

country 1
 - airport 1
    - departures
      - page 1
      - page ...
    - arrivals
      - page 1
      - page ...
- airport 2
    - departures
      - page 1
      - page ...
    - arrivals
      - page 
      - page ...
...

我使用两种方法来计算时间戳和URL查询：

def compute_timestamp(self):
    from datetime import datetime, date
    import calendar
    # +/- 24 heures
    d = date(2017, 4, 27)
    timestamp = calendar.timegm(d.timetuple())
    return timestamp
def build_api_call(self,code,page,timestamp):
    return 'https://api.flightradar24.com/common/v1/airport.json?code={code}&plugin[]=&plugin-setting[schedule][mode]=&plugin-setting[schedule][timestamp]={timestamp}&page={page}&limit=100&token='.format(
        code=code, page=page, timestamp=timestamp)

我将结果存储到CountryItem中，其中包含大量AirportItem到机场。我的item.py是：

class CountryItem(scrapy.Item):
    name = scrapy.Field()
    link = scrapy.Field()
    num_airports = scrapy.Field()
    airports = scrapy.Field()
    other_url= scrapy.Field()
    last_updated = scrapy.Field(serializer=str)
class AirportItem(scrapy.Item):
    name = scrapy.Field()
    code_little = scrapy.Field()
    code_total = scrapy.Field()
    lat = scrapy.Field()
    lon = scrapy.Field()
    link = scrapy.Field()
    departures = scrapy.Field()
    arrivals = scrapy.Field()

我的主要解析为所有国家建立一个国家项目(例如，我将其限制在以色列(。接下来，我屈服于每个国家/地区的scrapy.Request来刮擦机场。

###################################
# MAIN PARSE
####################################
def parse(self, response):
    count_country = 0
    countries = []
    for country in response.xpath('//a[@data-country]'):
        item = CountryItem()
        url =  country.xpath('./@href').extract()
        name = country.xpath('./@title').extract()
        item['link'] = url[0]
        item['name'] = name[0]
        item['airports'] = []
        count_country += 1
        if name[0] == "Israel":
            countries.append(item)
            self.logger.info("Country name : %s with link %s" , item['name'] , item['link'])
            yield scrapy.Request(url[0],meta={'my_country_item':item}, callback=self.parse_airports)

此方法为每个机场刮擦信息，还请每个机场用机场URL的scrapy.request刮擦出发和到达：

  ###################################
# PARSE EACH AIRPORT
####################################
def parse_airports(self, response):
    item = response.meta['my_country_item']
    item['airports'] = []
    for airport in response.xpath('//a[@data-iata]'):
        url = airport.xpath('./@href').extract()
        iata = airport.xpath('./@data-iata').extract()
        iatabis = airport.xpath('./small/text()').extract()
        name = ''.join(airport.xpath('./text()').extract()).strip()
        lat = airport.xpath("./@data-lat").extract()
        lon = airport.xpath("./@data-lon").extract()
        iAirport = AirportItem()
        iAirport['name'] = self.clean_html(name)
        iAirport['link'] = url[0]
        iAirport['lat'] = lat[0]
        iAirport['lon'] = lon[0]
        iAirport['code_little'] = iata[0]
        iAirport['code_total'] = iatabis[0]
        item['airports'].append(iAirport)
    urls = []
    for airport in item['airports']:
        json_url = self.build_api_call(airport['code_little'], 1, self.compute_timestamp())
        urls.append(json_url)
    if not urls:
        return item
    # start with first url
    next_url = urls.pop()
    return scrapy.Request(next_url, self.parse_schedule, meta={'airport_item': item, 'airport_urls': urls, 'i': 0})

使用递归方法parse_schedule，我将每个机场添加到国家项目中。因此，成员已经在这一点上为我提供了帮助。

###################################
# PARSE EACH AIRPORT OF COUNTRY
###################################
def parse_schedule(self, response):
        """we want to loop this continuously to build every departure and arrivals requests"""
        item = response.meta['airport_item']
        i = response.meta['i']
        urls = response.meta['airport_urls']
        urls_departures, urls_arrivals = self.compute_urls_by_page(response, item['airports'][i]['name'], item['airports'][i]['code_little'])
        print("urls_departures = ", len(urls_departures))
        print("urls_arrivals = ", len(urls_arrivals))
        ## YIELD NOT CALLED
        yield scrapy.Request(response.url, self.parse_departures_page, meta={'airport_item': item, 'page_urls': urls_departures, 'i':0 , 'p': 0}, dont_filter=True)
        # now do next schedule items
        if not urls:
            yield item
            return
        url = urls.pop()
        yield scrapy.Request(url, self.parse_schedule, meta={'airport_item': item, 'airport_urls': urls, 'i': i + 1})

self.compute_urls_by_page方法计算正确的URL，以检索一个机场的所有出发和到达。

###################################
# PARSE EACH DEPARTURES / ARRIVALS
###################################
def parse_departures_page(self, response):
    item = response.meta['airport_item']
    p = response.meta['p']
    i = response.meta['i']
    page_urls = response.meta['page_urls']
    print("PAGE URL = ", page_urls)
    if not page_urls:
        yield item
        return
    page_url = page_urls.pop()
    print("GET PAGE FOR  ", item['airports'][i]['name'], ">> ", p)
    jsonload = json.loads(response.body_as_unicode())
    json_expression = jmespath.compile("result.response.airport.pluginData.schedule.departures.data")
    item['airports'][i]['departures'] = json_expression.search(jsonload)
    yield scrapy.Request(page_url, self.parse_departures_page, meta={'airport_item': item, 'page_urls': page_urls, 'i': i, 'p': p + 1})

接下来，parse_schedule中的第一个收益率通常称为self.parse_departure_page递归方法会产生奇怪的结果。 scrapy致电此方法，但是我收集了一个机场的出发页面，我不明白为什么... 我的请求中可能有订购错误或产量源代码，所以也许您可以提供帮助我找出答案。

完整的代码在github上https://github.com/idees-rouen/flight-scrapping/tree/master/master/flight/flight/flight_project

您可以使用scrapy cawl airports命令运行它。

更新1：

我尝试使用yield from独自回答问题，而您可以看到答案底部...因此，如果您有想法？

Answer 1

是的，我终于在这里找到了答案...

使用递归yield时，需要使用yield from。这里简化了一个示例：

airport_list = ["airport1", "airport2", "airport3", "airport4"]
def parse_page_departure(airport, next_url, page_urls):
    print(airport, " / ", next_url)

    if not page_urls:
        return
    next_url = page_urls.pop()
    yield from parse_page_departure(airport, next_url, page_urls)
###################################
# PARSE EACH AIRPORT OF COUNTRY
###################################
def parse_schedule(next_airport, airport_list):
    ## GET EACH DEPARTURE PAGE
    departures_list = ["p1", "p2", "p3", "p4"]
    next_departure_url = departures_list.pop()
    yield parse_page_departure(next_airport,next_departure_url, departures_list)
    if not airport_list:
        print("no new airport")
        return
    next_airport_url = airport_list.pop()
    yield from parse_schedule(next_airport_url, airport_list)
next_airport_url = airport_list.pop()
result = parse_schedule(next_airport_url, airport_list)
for i in result:
    print(i)
    for d in i:
        print(d)

更新，请勿使用真实程序：

我尝试在此处使用真实程序来重现相同的yield from模式，但是我在scrapy.Request上使用它有一个错误，不明白为什么...

在这里python追溯：

Traceback (most recent call last):
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/reyman/Projets/Flight-Scrapping/flight/flight_project/spiders/AirportsSpider.py", line 209, in parse_schedule
    yield from scrapy.Request(url, self.parse_schedule, meta={'airport_item': item, 'airport_urls': urls, 'i': i + 1})
TypeError: 'Request' object is not iterable
2017-06-27 17:40:50 [scrapy.core.engine] INFO: Closing spider (finished)
2017-06-27 17:40:50 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

Answer 2

注释：...不完全清楚...您致电airportdata(响应，1(...在这里也有一些错字：self.pprint(schepen(

我使用class AirportData实现(限制为2页和2个航班(。
更新了我的代码，删除了class AirportData 并添加了class Page。
现在应该满足所有依赖性。

这是不是 typo，self.pprint(...是用于漂亮打印对象的CC_19，就像末尾显示的输出一样。我已经增强了class Schedule以显示基本用法。

---

评论：您的答案中的机场数据是什么？

编辑：删除class AirportData。
如# ENDPOINT上所述，用于page.arrivals和page.departures的飞行数据的Page object。(限制为2页和2班(

Page = [Flight 1, Flight 1, ... Flight n] 
schedule.airport['arrivals'] == [Page 1, Page 2, ..., Page n]
schedule.airport['departures'] == [Page 1, Page 2, ..., Page n]

---

注释：...我们有包含倍数的倍数页/到达。

是的，在第一个答案时，我没有任何api json响应以进一步。
现在，我从api json获得了响应，但没有反映给定的timestamp，从current date返回。api params看起来不常见，您有链接到描述吗？

---

尽管如此，请考虑这种简化的方法：

＃page对象持有到达/出发数据

class Page(object):
    def __init__(self, title, schedule):
        # schedule includes ['arrivals'] or ['departures]
        self.current = schedule['page']['current']
        self.total = schedule['page']['total']
        self.header = '{}:page:{} item:{}'.format(title, schedule['page'], schedule['item'])
        self.flight = []
        for data in schedule['data']:
            self.flight.append(data['flight'])
    def __iter__(self):
        yield from self.flight

＃计划对象持有一个机场所有页面

class Schedule(object):
    def __init__(self):
        self.country = None
        self.airport = None
    def __str__(self):
        arrivals = self.airport['arrivals'][0]
        departures = self.airport['departures'][0]
        return '{}nt{}ntt{}nttt{}ntt{}nttt{}'. 
            format(self.country['name'],
                   self.airport['name'],
                   arrivals.header,
                   arrivals.flight[0]['airline']['name'],
                   departures.header,
                   departures.flight[0]['airline']['name'], )

＃parse国家的每个机场

def parse_schedule(self, response):
    meta = response.meta
    if 'airport' in meta:
        # First call from parse_airports
        schedule = Schedule()
        schedule.country = response.meta['country']
        schedule.airport = response.meta['airport']
    else:
        schedule = response.meta['schedule']
    data = json.loads(response.body_as_unicode())
    airport = data['result']['response']['airport']
    schedule.airport['arrivals'].append(Page('Arrivals', airport['pluginData']['schedule']['arrivals']))
    schedule.airport['departures'].append(Page('Departures', airport['pluginData']['schedule']['departures']))
    page = schedule.airport['departures'][-1]
    if page.current < page.total:
        json_url = self.build_api_call(schedule.airport['code_little'], page.current + 1, self.compute_timestamp())
        yield scrapy.Request(json_url, meta={'schedule': schedule}, callback=self.parse_schedule)
    else:
        # ENDPOINT Schedule object holding one Airport.
        # schedule.airport['arrivals'] and schedule.airport['departures'] ==
        #   List of Page with List of Flight Data
        print(schedule)

＃解析每个机场

def parse_airports(self, response):
    country = response.meta['country']
    for airport in response.xpath('//a[@data-iata]'):
        name = ''.join(airport.xpath('./text()').extract()[0]).strip()
        if 'Charles' in name:
            meta = response.meta
            meta['airport'] = AirportItem()
            meta['airport']['name'] = name
            meta['airport']['link'] = airport.xpath('./@href').extract()[0]
            meta['airport']['lat'] = airport.xpath("./@data-lat").extract()[0]
            meta['airport']['lon'] = airport.xpath("./@data-lon").extract()[0]
            meta['airport']['code_little'] = airport.xpath('./@data-iata').extract()[0]
            meta['airport']['code_total'] = airport.xpath('./small/text()').extract()[0]
            json_url = self.build_api_call(meta['airport']['code_little'], 1, self.compute_timestamp())
            yield scrapy.Request(json_url, meta=meta, callback=self.parse_schedule)

＃主解析

注意： response.xpath('//a[@data-country]')返回 asl countrys 两次！

def parse(self, response):
    for a_country in response.xpath('//a[@data-country]'):
            name = a_country.xpath('./@title').extract()[0]
            if name == "France":
                country = CountryItem()
                country['name'] = name
                country['link'] = a_country.xpath('./@href').extract()[0]
                yield scrapy.Request(country['link'],
                                     meta={'country': country},
                                     callback=self.parse_airports)

qutput ：缩短到 2 页面和 2 每页飞行

France
    Paris Charles de Gaulle Airport
        Departures:(page=(1, 1, 7)) 2017-07-02 21:28:00 page:{'current': 1, 'total': 7} item:{'current': 100, 'limit': 100, 'total': 696}
            21:30 PM    AF1558  Newcastle Airport (NCL) Air France ARJ  Estimated dep 21:30
            21:30 PM    VY8833  Seville San Pablo Airport (SVQ) Vueling 320 Estimated dep 21:30
            ... (omitted for brevity)
        Departures:(page=(2, 2, 7)) 2017-07-02 21:28:00 page:{'current': 2, 'total': 7} item:{'current': 100, 'limit': 100, 'total': 696}
            07:30 AM    AF1680  London Heathrow Airport (LHR)   Air France 789  Scheduled
            07:30 AM    SN3628  Brussels Airport (BRU)  Brussels Airlines 733   Scheduled
            ... (omitted for brevity)
        Arrivals:(page=(1, 1, 7)) 2017-07-02 21:28:00 page:{'current': 1, 'total': 7} item:{'current': 100, 'limit': 100, 'total': 693}
            16:30 PM    LY325   Tel Aviv Ben Gurion International Airport (TLV) El Al Israel Airlines B739  Estimated 21:29
            18:30 PM    AY877   Helsinki Vantaa Airport (HEL)   Finnair E190    Landed 21:21
            ... (omitted for brevity)
        Arrivals:(page=(2, 2, 7)) 2017-07-02 21:28:00 page:{'current': 2, 'total': 7} item:{'current': 100, 'limit': 100, 'total': 693}
            00:15 AM    AF982   Douala International Airport (DLA)  Air France 772  Scheduled
            23:15 PM    AA44    New York John F. Kennedy International Airport (JFK)    American Airlines B763  Scheduled
            ... (omitted for brevity)

用Python测试：3.4.2- scrapy 1.4.0

Answer 3

我尝试在本地克隆并进行更好的调查，但是当它到达出发时，我得到了一些连接的错误，因此我不确定我的建议答案是否会解决它，无论如何：

###################################
# PARSE EACH AIRPORT OF COUNTRY
###################################
def parse_schedule(self, response):
    """we want to loop this continuously to build every departure and arrivals requests"""
    item = response.meta['airport_item']
    i = response.meta['i']
    urls = response.meta['airport_urls']
    urls_departures, urls_arrivals = self.compute_urls_by_page(response, item['airports'][i]['name'], item['airports'][i]['code_little'])
    if 'urls_departures' in response.meta:
        urls_departures += response.meta["urls_departures"]
    if 'urls_arrivals' in response.meta:
        urls_arrivals += response.meta["urls_arrivals"]
    print("urls_departures = ", len(urls_departures))
    print("urls_arrivals = ", len(urls_arrivals))
    item['airports'][i]['departures'] = []
    # now do next schedule items
    if not urls:
        yield scrapy.Request(urls_departures.pop(), self.parse_departures_page, meta={'airport_item': item, 'page_urls': urls_departures, 'i':i , 'p': 0}, dont_filter=True)
    else:
        url = urls.pop()
        yield scrapy.Request(url, self.parse_schedule, meta={'airport_item': item, 'airport_urls': urls, 'i': i + 1, 'urls_departures': urls_departures, 'urls_arrivals': urls_arrivals})
###################################
# PARSE EACH DEPARTURES / ARRIVALS
###################################
def parse_departures_page(self, response):
    item = response.meta['airport_item']
    p = response.meta['p']
    i = response.meta['i']
    page_urls = response.meta['page_urls']
    jsonload = json.loads(response.body_as_unicode())
    json_expression = jmespath.compile("result.response.airport.pluginData.schedule.departures.data")
    # Append a new page
    item['airports'][i]['departures'].append(json_expression.search(jsonload))
    if len(page_urls) > 0:
        page_url = page_urls.pop()
        yield scrapy.Request(page_url, self.parse_departures_page, meta={'airport_item': item, 'page_urls': page_urls, 'i': i, 'p': p + 1}, dont_filter=True)
    else:
        yield item

，但基本上这些是您的错误：

1. 在您的parse_schedule中，在您的parse_departures_page中，您有最终项目的条件；

2. 您将错误的URL传递给Parse_departures_page;

3. 您需要dont_filter =在parse_departures_page;

上

4. 您正在尝试保留大量循环以将更多信息解析到同一对象

我提出的更改将跟踪该机场上的所有urls_departures，以便您可以迭代然后在parse_departures_page上解决问题。

即使解决了您的问题，我确实建议您更改数据结构，以便您可以有多个出发的项目并能够更有效地提取此信息。