我正在运行一组服务,这些服务使用redis作为队列、代理。像芹菜。
当我启动容器(使用docker-compose)时,一切都很好,但过了"一段时间",试图连接到redis的服务会抛出Timeout异常。
这是我的redis容器日志:
1:C 09 Jan 15:10:30.407 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 09 Jan 15:10:30.408 # Redis version=4.0.11, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 09 Jan 15:10:30.408 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 09 Jan 15:10:30.411 * Running mode=standalone, port=6379.
1:M 09 Jan 15:10:30.411 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 09 Jan 15:10:30.411 # Server initialized
1:M 09 Jan 15:10:30.411 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 09 Jan 15:10:30.412 * Ready to accept connections
1:M 09 Jan 16:10:48.695 * 1 changes in 3600 seconds. Saving...
1:M 09 Jan 16:10:48.731 * Background saving started by pid 19
19:C 09 Jan 16:10:48.738 * DB saved on disk
19:C 09 Jan 16:10:48.739 * RDB: 0 MB of memory used by copy-on-write
1:M 09 Jan 16:10:48.833 * Background saving terminated with success
我使用的是默认配置,这是它的docker组成部分:
redis:
image: redis:latest
这是我从烧瓶/芹菜容器中得到的例外:
kombu.exceptions.OperationalError: Timeout connecting to server
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/env/lib/python3.6/site-packages/celery/beat.py", line 241, in apply_entry
result = self.apply_async(entry, producer=producer, advance=False)
File "/app/env/lib/python3.6/site-packages/celery/beat.py", line 358, in apply_async
entry, exc=exc)), sys.exc_info()[2])
File "/app/env/lib/python3.6/site-packages/vine/five.py", line 178, in reraise
raise value.with_traceback(tb)
File "/app/env/lib/python3.6/site-packages/celery/beat.py", line 350, in apply_async
**entry.options)
File "/app/env/lib/python3.6/site-packages/celery/app/task.py", line 535, in apply_async
**options
File "/app/env/lib/python3.6/site-packages/celery/app/base.py", line 745, in send_task
amqp.send_task_message(P, name, message, **options)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/app/env/lib/python3.6/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
sys.exc_info()[2])
File "/app/env/lib/python3.6/site-packages/vine/five.py", line 178, in reraise
raise value.with_traceback(tb)
File "/app/env/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
yield
File "/app/env/lib/python3.6/site-packages/celery/app/base.py", line 744, in send_task
self.backend.on_task_call(P, task_id)
File "/app/env/lib/python3.6/site-packages/celery/backends/redis.py", line 265, in on_task_call
self.result_consumer.consume_from(task_id)
File "/app/env/lib/python3.6/site-packages/celery/backends/redis.py", line 126, in consume_from
self._consume_from(task_id)
File "/app/env/lib/python3.6/site-packages/celery/backends/redis.py", line 132, in _consume_from
self._pubsub.subscribe(key)
File "/app/env/lib/python3.6/site-packages/redis/client.py", line 2482, in subscribe
ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
File "/app/env/lib/python3.6/site-packages/redis/client.py", line 2404, in execute_command
self._execute(connection, connection.send_command, *args)
File "/app/env/lib/python3.6/site-packages/redis/client.py", line 2408, in _execute
return command(*args)
File "/app/env/lib/python3.6/site-packages/redis/connection.py", line 610, in send_command
self.send_packed_command(self.pack_command(*args))
File "/app/env/lib/python3.6/site-packages/redis/connection.py", line 585, in send_packed_command
self.connect()
File "/app/env/lib/python3.6/site-packages/redis/connection.py", line 486, in connect
raise TimeoutError("Timeout connecting to server")
可能出了什么问题?
如果我永远无法连接,我理解,但问题会在一段时间后开始,它会永远返回超时,直到我重新启动容器。
谢谢!
问题是iptables规则在docker规则和其他一些规则之间出现了混乱。我修复的方法是使用Nginx作为反向代理(网关模式),并在其上管理访问规则,而不是使用iptables。
这样,一切都如预期的那样运转。
谢谢!