单个会话多个帖子/获取Python请求



我正在尝试编写一个使用Python请求模块自动下载一些文件的爬行者。但是,我遇到了一个问题。

我初始化了一个新的请求会话,然后我使用Post方法登录网站,此后,只要我尝试使用post/get方法(以下简化的代码):

s=requests.session()
s.post(url,data=post_data, headers=headers)
#up to here everything is correct, the next step will report error 
s.get(url) or s.post(url) even repeat s.post(url,data=post_data, headers=headers) will report error 

它将按照下面的错误报告错误:

Traceback (most recent call last):
File"/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 374, in _make_request
httplib_response = conn.getresponse()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1162, in getresponse
raise ResponseNotReady(self.__state)
http.client.ResponseNotReady: Request-sent
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/util/retry.py", line 245, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/packages/six.py", line 309, in reraise
raise value.with_traceback(tb)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 374, in _make_request
httplib_response = conn.getresponse()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1162, in getresponse
raise ResponseNotReady(self.__state)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ResponseNotReady('Request-sent',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 280, in <module>
test()
File "test.py", line 273, in test
emuch1.getEbook()
File "test.py", line 146, in getEbook
self.downloadEbook(ebook)
File "test.py", line 179, in downloadEbook
file_url=self.downloadEbookGetFileUrl(ebook).decode('gbk')
File "test.py", line 211, in downloadEbookGetFileUrl
download_url=self.downloadEbookGetUrl(ebook)
File "test.py", line 200, in downloadEbookGetUrl
respond_ebook=self.session.get(ebook_url)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 477, in get
return self.request('GET', url, **kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ResponseNotReady('Request-sent',))

我完全不知道为什么会发生这种情况,有人可以帮我吗?

请求使用URLLIB3的内部版本。我的印象是,内部Urllib3之间有一个版本不匹配并请求本身。

httplib_response = conn.getResponse(buffering = true)typeerror: getResponse()有一个意外的关键字参数'buffering'

似乎表明请求正在调用urllib3(内部版本,而不是python的),但要指定不存在的"缓冲"。

其他问题类似于我的经历。

最新的请求版本(2.6。*)还有其他问题正在解决。我怀疑您正在使用该版本。尝试倒退以前的2.4.1版,甚至是2.2.1。如果您指定,可以在程序的顶部,您要使用哪个版本:

__requires__ = ["requests==2.2.1"]
import pkg_resources

(至少在导入请求本身之前)

解决方案:上周,我与开发团队交换了几封邮件,看起来他们在修复程序方面已经生产了2.7!(实际上,我看到它仅在昨天上传到)。因此,如果您遇到类似的问题,请下载最新版本!

通过将请求升级到最新版本来解决该问题,它可能是2.6中的错误(也许是此版本,不确定)。

最新更新