如何在 SCRAPY 中处理对中间件的多个请求(验证码和多次重试)

Question

我正在尝试构建一个仅用 Scrapy 破坏动态验证码的蜘蛛，我已经做到了，但当然当我破坏验证码时并不总是正确的，所以我必须让它重试多次（最多 10 次）才能真正进入多个帐户的"登录"页面（顺便说一句，我正在使用 PIL 和 pytesseract 来处理验证码）。我已经阅读了如何设置 Scrapy 来处理验证码，这是一个很大的帮助，但它给了我这个错误：

TypeError: process_response() got multiple values for argument 'request'

这是我的蜘蛛代码：

#self.crawler.settings['MY_ACCOUNTS']= i have defined my list of accounts with "keys"
def start_requests(self):
    requests = []
    for i, account in enumerate(self.crawler.settings['MY_ACCOUNTS']):
        request = scrapy.Request(same_url, callback = self.parse, meta={'key':account['key']} ,dont_filter=True)
        request.meta['cookiejar'] = i
        requests.append(request)
        yield request
#I HAVE TO USE MULTIPLE REQUEST JUST TO GET THE CORRECT SESSION FOR THE COOKIES AND GET THE IMAGE, IF NOT THE IMAGE ALWAYS CHANGE
def parse(self, response):
    key= response.meta.get('key')
    sessionid = response.headers.getlist('Set-Cookie')[0].decode().split(';')[0]
    cookien = sessionid.split('=')[0].strip()
    sessionid = sessionid.split('=')[1].strip()
    yield scrapy.Request(same_url, meta = {'cookieN': cookien, 'cookieV' : sessionid, 'key':key,'cookiejar': response.meta['cookiejar']},cookies = {cookien :sessionid}, callback = self.parse_with_cookies, dont_filter=True)
#HERE I'M TRYING TO BREAK THE CAPTCHA FILLING A FORM
def parse_with_cookies(self, response):
    #break captcha (get text)
    data = {'key':key,
    'catcha':text
    }#fill with the data
    yield FormRequest.from_response(response=response,formname="mainForm", formdata=data, callback = self.parse_each_key,meta = {'solve_captcha': True,'CookieName':cookieN,'CookieValue':cookieV , 'key':key,'cookiejar': response.meta['cookiejar'],'catpcha_retries':0},cookies={cookieN:cookieV},dont_filter = True)
#the status code is always 200 even if the captcha is not broken so i search for words if it's incorrect
def parse_each_key(self, response):
    if ("incorrect" not in response.text) and ("error" not in response.text):
         #extract what i want
    else:
         self.logger.error("Login failed")

-----中间件

---

#i haven't change to much from the link, i know that i have to fill the functions find_catpcha and solve_captcha but it doesn't even get there
class CaptchaMiddleware(object):
max_retries = 5
def process_response(request, response, spider):
    if not request.meta.get('solve_captcha', False):
        return response  # only solve requests that are marked with meta key
    catpcha = find_catpcha(response)
    solved = solve_captcha(captcha)
    if solved:
        response.meta['catpcha'] = captcha
        response.meta['solved_catpcha'] = solved
        return response
    else:
        # retry page for new captcha
        # prevent endless loop
        if request.meta.get('catpcha_retries', 0) == 5:
            logging.warning('max retries for captcha reached for {}'.format(request.url))
            raise IgnoreRequest 
        request.meta['dont_filter'] = True
        request.meta['captcha_retries'] = request.meta.get('captcha_retries', 0) + 1
        return request

我不知道这是因为我同时发送多个请求还是代码中的某些内容。

谢谢

Answer 1

中间件中process_response()的定义中缺少self参数。这就是异常的原因。

如果将process_response(request, response, spider)替换为process_response(self, request, response, spider)则应停止获取该异常。