您如何处理芹菜(不是您的代码)引发的异常?



所以现在在我的烧瓶应用程序中,我正在使用 Celery 在远程机器上部署服务器。现在,我有一个枚举 status ,它指示我的部署过程的生命周期:

@celery.task(bind=True)
def deploy_server(self, server_id):
server = Server.query.get(server_id)
if not server.can_launch():
return
try:
server.status = RemoteStatus.LAUNCHING
db.session.commit()
verify_DNS(server)
host = server.server.ssh_user + '@' + server.server.ip
execute(fabric_deploy_server, self, server, hosts=host)
server.status = RemoteStatus.LAUNCHED
db.session.commit()
except Exception as e:
server.status = RemoteStatus.ERROR
db.session.commit()
traceback.print_exc()
raise e

如您所见,部署服务器时,其状态更改为"正在启动"。如果存在异常,它将更改为错误。

我发现了一个完全绕过这组代码的异常:当我用太多请求使我的芹菜服务器过载时,我得到这个异常:

[2017-07-09 18:00:03,127: WARNING/PoolWorker-3] /app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: ConnectionError('max number of clients reached',): 
Traceback (most recent call last): 
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 427, in trace_task 
uuid, retval, task_request, publish_result, 
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 152, in mark_as_done 
self.store_result(task_id, result, state, request=request) 
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result 
request=request, **kwargs) 
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 652, in _store_result 
self.set(self.get_key_for_task(task_id), self.encode(meta)) 
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 204, in set 
return self.ensure(self._set, (key, value), **retry_policy) 
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 194, in ensure 
**retry_policy) 
File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time 
return fun(*args, **kwargs) 
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 213, in _set 
pipe.execute() 
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute 
return execute(conn, stack, raise_on_error) 
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction 
connection.send_packed_command(all_cmds) 
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command 
self.connect() 
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 446, in connect 
self.on_connect() 
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 514, in on_connect 
if nativestr(self.read_response()) != 'OK': 
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 577, in read_response 
response = self._parser.read_response() 
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 255, in read_response 
raise error 
ConnectionError: max number of clients reached 
exc, exc_info.traceback))) 

我最大的问题是这个错误是在我的 Try/Catch 集团之外的某个地方引发的。因此,当发生此异常时,我的所有服务器都保持"启动"模式而不是"错误"。

如何捕获此异常并修改服务器状态?

在 redis 2.4 中,最大连接数的硬编码限制为 10,000。在 redis 2.6+ 中,您可以在 redis.conf 中指定最大客户端数。此外,这不是您的经纪人 Redis 拒绝接受连接的芹菜问题,这就是问题所在。

使用 redisCLI 设置可由 redis 同时处理的最大客户端数。查看 Redis 客户端

最新更新