我正在测试刮擦和飞溅。我有一只蜘蛛:
class MySpider(scrapy.Spider):
# start_urls = ["http://yahoo.com"]
name = 'mytest'
def __init__(self, state='CA', city='San_Francisco', *args, **kwargs):
super().__init__(*args, **kwargs)
self.state = state
self.city = city
self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=state, city=city)]
# self.le = LinkExtractor(allow=r'^https://www.trulia.com/property')
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 7.5},)
如果我摆脱了 init 函数并取消注释 strt-urls 行,蜘蛛就会工作。 逐步执行它的问题出现在:
self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=state, city=city)]
追踪:
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider opened
2019-06-23 09:15:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-06-23 09:15:00 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method SplashMiddleware.spider_opened of <scrapy_splash.middleware.SplashMiddleware object at 0x
00000292FF8106A0>>
Traceback (most recent call last):
File " libsite-packagestwistedinternetdefer.py", line 151, in maybeDeferred
result = f(*args, **kw)
File "....libsite-packagespydispatchrobustapply.py", line 55, in robustApply
return receiver(*arguments, **named)
File "....libsite-packagesscrapy_splashmiddleware.py", line 245, in spider_opened
spider.state.setdefault(self.remote_keys_key, {})
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-06-23 09:15:00 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
File "....libsite-packagesscrapycoreengine.py", line 127, in _next_request
request = next(slot.start_requests)
File "....libsite-packagesscrapy_splashmiddleware.py", line 165, in process_start_requests
spider.state.setdefault(self.local_values_key, {}) # fingerprint => value dict
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider opened
2019-06-23 09:15:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-06-23 09:15:00 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method SplashMiddleware.spider_opened of <scrapy_splash.middleware.SplashMiddleware object at 0x
00000292FF8106A0>>
Traceback (most recent call last):
File " libsite-packagestwistedinternetdefer.py", line 151, in maybeDeferred
result = f(*args, **kw)
File "....libsite-packagespydispatchrobustapply.py", line 55, in robustApply
return receiver(*arguments, **named)
File "....libsite-packagesscrapy_splashmiddleware.py", line 245, in spider_opened
spider.state.setdefault(self.remote_keys_key, {})
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-06-23 09:15:00 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
File "....libsite-packagesscrapycoreengine.py", line 127, in _next_request
request = next(slot.start_requests)
File "....libsite-packagesscrapy_splashmiddleware.py", line 165, in process_start_requests
spider.state.setdefault(self.local_values_key, {}) # fingerprint => value dict
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Closing spider (finished)
2019-06-23 09:15:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 838859),
'log_count/ERROR': 2,
'log_count/INFO': 9,
'start_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 822180)}
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider closed (finished)
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Closing spider (finished)
2019-06-23 09:15:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 838859),
'log_count/ERROR': 2,
'log_count/INFO': 9,
'start_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 822180)}
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider closed (finished)
问题是当你查看抓取的中间件代码时,你的参数名称state
https://github.com/scrapy-plugins/scrapy-splash/blob/e40ca4f3b367ab463273bee1357d3edfe0601f0d/scrapy_splash/middleware.py#L245
state
对它有特殊的含义,它使用相同的dict
。所以你需要做的只是重命名参数
class MySpider(scrapy.Spider):
# start_urls = ["http://yahoo.com"]
name = 'mytest'
def __init__(self, STATE='CA', CITY='San_Francisco', *args, **kwargs):
super().__init__(*args, **kwargs)
self.state = state
self.city = city
self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=STATE, city=CITY)]
# self.le = LinkExtractor(allow=r'^https://www.trulia.com/property')
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 7.5},)