asyncio:等待执行器运行的长任务完成时的异常处理



我有以下代码段由python 3.10.5运行:

import time
import asyncio
async def main():
loop = asyncio.get_running_loop()
loop.run_in_executor(None, blocking)
print(f"{time.ctime()} Hello!")
await asyncio.sleep(1.0)
print(f"{time.ctime()} Goodbye!")
def blocking():
time.sleep(5.0)
print(f"{time.ctime()} Hello from thread!")

try:
asyncio.run(main())
except KeyboardInterrupt:
print("Cancelled.")

当我让它运行时,它会优雅地退出,因为python 3.9中添加了shutdown_default_executor()方法,通过将此任务封装在协程中,可以解决在执行器中运行的任务比主事件循环更持久的问题。所以我有以下输出:

Sun Sep 11 19:04:25 2022 Hello!
Sun Sep 11 19:04:26 2022 Goodbye!
Sun Sep 11 19:04:30 2022 Hello from thread!

接下来,当我在第一行输出后按下Ctrl-C时,我得到:

Sun Sep 11 19:04:42 2022 Hello!
^CSun Sep 11 19:04:47 2022 Hello from thread!
Cancelled.

因此,它仍然能够处理这种情况。但当我在Goodbye!行之后执行时(当主协同程序已经完成,我正在等待执行器中的任务完成时(,我得到:

Sun Sep 11 19:04:49 2022 Hello!
Sun Sep 11 19:04:50 2022 Goodbye!
^CCancelled.
Sun Sep 11 19:04:54 2022 Hello from thread!
exception calling callback for <Future at 0x7f58475183d0 state=finished returned NoneType>
Traceback (most recent call last):
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 330, in _invoke_callbacks
callback(self)
File "/usr/lib/python3.10/asyncio/futures.py", line 398, in _call_set_state
dest_loop.call_soon_threadsafe(_set_state, destination, source)
File "/usr/lib/python3.10/asyncio/base_events.py", line 795, in call_soon_threadsafe
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception in thread Thread-1 (_do_shutdown):
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/base_events.py", line 576, in _do_shutdown
self.call_soon_threadsafe(future.set_result, None)
File "/usr/lib/python3.10/asyncio/base_events.py", line 795, in call_soon_threadsafe
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/asyncio/base_events.py", line 578, in _do_shutdown
self.call_soon_threadsafe(future.set_exception, ex)
File "/usr/lib/python3.10/asyncio/base_events.py", line 795, in call_soon_threadsafe
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

问题是,为什么我在这里遇到运行时错误,但当我在Hello!行之后点击Ctrl-C时却设法避免了它(第二种情况(?如何优雅地处理此运行时错误?

我想我已经想通了,但请小心——我对asyncio还很陌生。

我基于python 3.10异步.run代码:

def run(main, *, debug=None):
# More code here
# ...
try:
events.set_event_loop(loop)
if debug is not None:
loop.set_debug(debug)
return loop.run_until_complete(main)
finally:
try:
_cancel_all_tasks(loop)
loop.run_until_complete(loop.shutdown_asyncgens())
loop.run_until_complete(loop.shutdown_default_executor())
finally:
events.set_event_loop(None)
loop.close()

您所描述的行为是由那些嵌套的try-filly块来解释的。

如果KeyboardInterrupt发生在loop.run_until_complete(main)期间(在您的情况下是在Goodbye!之前(,则执行内部try-finally,从而正确地使用loop.shutdown_default_executor()处理blocking

另一方面,如果异常发生在Goodbye!之后,则当前执行的代码是loop.shutdown_default_executor(),并且由于进一步的清理在不等待任何东西的情况下关闭了循环,因此包含blockingFuture抛出RuntimeError('Event loop is closed')

Future似乎还在等待。。。除非你再次点击Ctrl-C。然后

Exception ignored in: <module 'threading' from '/usr/lib/python3.10/threading.py'>
Exception in thread Thread-1 (_do_shutdown):
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1537, in _shutdown
File "/usr/lib/python3.10/asyncio/base_events.py", line 576, in _do_shutdown
self.call_soon_threadsafe(future.set_result, None)
File "/usr/lib/python3.10/asyncio/base_events.py", line 795, in call_soon_threadsafe
atexit_call()
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
t.join()
File "/usr/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
raise RuntimeError('Event loop is closed')
File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
RuntimeError: Event loop is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
if lock.acquire(block, timeout):
KeyboardInterrupt:

引发异常。

因此,似乎还有一层——这次是在线程级别——在等待执行器:D

无论如何,我认为这是意料之中的事——我们应该允许用户在不等待的情况下杀死程序(这可能需要很长时间(。

回到你的问题——如何优雅地处理它。我认为你做不到。错误被抛出到Future逻辑中,我们在stderr上看到的只是一个未使用结果的日志。但仅仅因为我们不能处理它并不意味着用户必须看到它

我不知道这是否是一个好的做法,但您可以将stderr重定向到null设备。或者弄清楚是哪个记录器给他打了补丁?适用于我的代码:

from contextlib import redirect_stderr
from os import devnull
fnull = open(devnull, 'w')
r = redirect_stderr(fnull)
r.__enter__()

最新更新