如何处理 urllib.error.URLError: <urlopen error [WinError 10054] 远程主机强行关闭了现有连接>?



我有以下一小段python脚本:

from urllib.request import urlopen
def download_file(url):
fp = open(url.split("=")[-1] + ".pdf", 'wb')
req = urlopen(url)
CHUNK = 20480
chunk = req.read(CHUNK)
fp.write(chunk)
fp.close()
for i in range(1, 10000, 1):
download_file("__some__url__" + str(i))
print("Done.")

我一直在运行这个脚本,但过了一段时间(比如下载100个文件后(,由于某种原因,它会出现错误:urllib.error.URLError: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host>

我如何修改我的代码来处理这个错误,即它不应该停止脚本和处理,即等待连接恢复,然后在它离开的地方继续下载?

附言:我知道它只从URL下载20KB。

对于此错误的可能原因,您可以查看python:[Erno 10054]远程主机强制关闭了现有连接。在阅读了最高投票率的答案后,我的结论是,这是可能发生的,你的代码应该做好准备。

我会在这里的循环中使用try: ... except ...块,并在重试失败的连接之前增加延迟:

def download_file(url):
# prefere with when exceptions are expected
with open(url.split("=")[-1] + ".pdf", 'wb') as fp:
delay = 5
max_retries = 3
for _ in range(max_retries):
try:
req = urlopen(url)
CHUNK = 20480
chunk = req.read(CHUNK)
fp.write(chunk)
break                # do not loop after a successfull download...
except urllib.error.URLError:
time.sleep(delay)
delay *= 2
else:                # signal an abort if download was not possible
print(f"Failed for {url} after {max_retries} attempt")
# or if you want to abort the script
# raise Exception(f"Failed for {url} after {max_retries} attempt")

相关内容

最新更新