如何在异常超时后重试,并从dict中删除



我想增加抓取内容的覆盖率,避免超时,但我希望如果页面出现超时错误,然后重试,但如果第二轮超时增加了超时计数器,则重试URL,最多重试3次,如果没有增加从dict 中删除

async def _worker(self, i):
while True:
url = await self.fetching.get()
if url is None:
return
#logger.info(f'Fetch worker {i} is fetching a URL: {url}')
try:
site = await self._fetch_rss(url)
await self._processing(site)
except asyncio.TimeoutError as err:
while self.timeouts[url] < 1:
logger.info("adding {}".format(url))
site = await self.fetch(url)
await self.process(site)
self.timeouts[url] += 1

通常;尝试n次,如果我从未成功,就会犯错";如下所示:

for _ in range(MAX_TRIES):
try:
# do something
pass
except Exception:
# handle exception
pass
else:
break
else: # A lesser-known syntax; else after for is only run if the for *wasn't* broken out of
# We never broke, which means we never had a success, so handle the failure here
pass

最新更新