我有一个仅部分执行的Django芹菜任务。
我启动应用程序,连接看起来不错:
info/mainprocess]连接到 redis://elasticache.cache.amazonaws.com:6379/0 [2018-02-17 23:27:24,314:info/mainprocess] mingle:寻找邻居 [2018-02-17 23:27:25,339:info/mainprocess] Mingle:一个人 [2018-02-17 23:27:25,604:Info/MainProcess] Worker1@test_vmstracker_com准备就绪。
我启动该过程,并收到任务:
[2018-02-17 23:27:49,810:info/mainprocess]收到任务: tracking.tasks.scalate [92F54D48202] ETA:[2018-02-18 07:27:59.797380 00:00] [2018-02-17 23:27:49,830:info/mainprocess] 收到的任务:tracking.tasks.scalate [09A0AEBEF72B] ETA:[2018-02-18 07:28:19.809712 00:00] [2018-02-17 23:28:00,205: 警告/forkpoolworker-7] - 我的应用程序正在工作 -
然后我开始遇到错误,并且无法完成我的应用发送电子邮件的任务
[2018-02-17 23:28:00,214:错误/forkpoolworker-7]连接到redis 丢失:重试(0/20(。[2018-02-17 23:28:00,220: 错误/forkpoolworker-7]连接到redis丢失:重试(1/2
有人知道为什么只执行执行,然后连接丢失?
这是完整的堆栈:
[2018-02-17 23:28:19,382: WARNING/ForkPoolWorker-7] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:549: RuntimeWarning: Exception raised outside body: ConnectionError("Error while reading from socket: ('Connection closed by server.',)",):
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 177, in _read_from_socket
raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2879, in execute
return execute(conn, stack, raise_on_error)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2764, in _execute_transaction
self.parse_response(connection, '_')
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2838, in parse_response
self, connection, command_name, **options)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 624, in read_response
response = self._parser.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 284, in read_response
response = self._buffer.readline()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 216, in readline
self._read_from_socket()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 191, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 177, in _read_from_socket
raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 434, in trace_task
uuid, retval, task_request, publish_result,
File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 152, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 309, in store_result
request=request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 652, in _store_result
self.set(self.get_key_for_task(task_id), self.encode(meta))
File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 213, in set
return self.ensure(self._set, (key, value), **retry_policy)
File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 203, in ensure
**retry_policy)
File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 222, in _set
pipe.execute()
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2894, in execute
return execute(conn, stack, raise_on_error)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2764, in _execute_transaction
self.parse_response(connection, '_')
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2838, in parse_response
self, connection, command_name, **options)
File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 624, in read_response
response = self._parser.read_response()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 284, in read_response
response = self._buffer.readline()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 216, in readline
self._read_from_socket()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 191, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
exc, exc_info.traceback)))
问题是我的应用服务器和经纪人实例太小。我正在使用EC2。一旦我升级到大型硬件,问题就消失了。在CPU,内存和网络方面,EC2实例或Elasticache实例太小。