我有以下会话对象:
import requests
from requests.adapters import Retry, HTTPAdapter
from requests.sessions import Session
retry_strategy: Retry = Retry(
total=5,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["HEAD", "GET", "OPTIONS", "POST"],
backoff_factor=1
)
http_adapter = HTTPAdapter(max_retries=retry_strategy)
session = requests.Session()
session.mount("https://", http_adapter)
偶尔服务器似乎会断开与RST TCP数据包的连接,导致以下错误:
{"isError": true,
"type": "ConnectionError",
"message": "[Errno 104] Connection reset by peer",
"traceback": "... omitted for brevity ...
File "/var/task/requests/sessions.py", line 590, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/var/task/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/var/task/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/var/task/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)"
- 是否可以在
Session
对象中注入Retry
对象,以便自动重试此特定的ConnectionError
- 是否有其他方法可以在没有任何额外自定义逻辑的情况下自动重试,例如将其封装在
try... except
语句中或创建自定义HTTPAdapter
- 我看过多篇关于
104 connection reset by peer errors
的帖子,但没有人问如何自动重试。为什么?我想我不会是唯一一个有这个问题的人吧
查看https://github.com/jd/tenacity
我通常会做这样的
from tenacity import wait_exponential
@retry(wait=wait_exponential(multiplier=1, min=4, max=10))
def wait_exponential_1():
# my request here
您可以创建一个函数,该函数产生一个用坚韧性包裹的会话。
我最终得到了退避:
@backoff.on_exception(
wait_gen=backoff.expo,
exception=requests.ConnectionError,
max_time=time_units.in_s.SEC_15,
)
def robust_session_request(*args, **kwargs) -> requests.Response:
return session.request(*args, **kwargs)
您需要提供;"连接";自变量:
from requests import adapters
import urllib3
sync_session = requests.Session()
retry_adapter = adapters.HTTPAdapter(max_retries=urllib3.Retry(total=1, connect=1))
session.mount('https://', retry_adapter)
session.mount('http://', retry_adapter)
测试:
class AdapterTest(unittest.TestCase):
@mock.patch.object(urllib3.connectionpool.HTTPConnectionPool, '_make_request',
side_effect=ConnectionResetError)
def test_adapter(self, mocked):
s = requests_extensions.HookableSession()
try:
s.mount('http://', adapters.HTTPAdapter(max_retries=urllib3.Retry(total=1, connect=1)))
s.request('GET', 'http://www.test.com')
except requests.ConnectionError:
pass
self.assertEqual(mocked.call_count, 2)
使用这个答案,我创建了一个返回"对等方重置连接";总是在一个终端上启动服务器:
$ python server.py
并在另一个终端上进行测试:
$ curl http://localhost:5500
curl: (56) Recv failure: Connection reset by peer
服务器很好地显示尝试了一个连接:
$ python server.py
new client from 127.0.0.1:55894
现在,在这个答案的基础上,让我们尝试使用带有Retry:的Python
>>> import requests
>>> import urllib3.util
>>> import requests.adapters
>>> s = requests.Session()
>>> retries = urllib3.util.Retry(connect=2, read=3, redirect=4, status=5, other=6)
>>> s.mount('http://', requests.adapters.HTTPAdapter(max_retries=retries))
>>> s.get("http://localhost:5500")
服务器显示了4次连接尝试(假设您在上面的curl测试后重新启动了它(——1次初始尝试和3次read
重试。
$ python server.py
new client from 127.0.0.1:60790
new client from 127.0.0.1:60798
new client from 127.0.0.1:60810
new client from 127.0.0.1:60818
您也可以只使用total
:
>>> import requests
>>> import urllib3.util
>>> import requests.adapters
>>> s = requests.Session()
>>> retries = urllib3.util.Retry(total=5)
>>> s.mount('http://', requests.adapters.HTTPAdapter(max_retries=retries))
>>> s.get("http://localhost:5500")
$ python server.py
new client from 127.0.0.1:37164
new client from 127.0.0.1:37176
new client from 127.0.0.1:37190
new client from 127.0.0.1:37192
new client from 127.0.0.1:42852
new client from 127.0.0.1:42864