Celery with MongoDB Broker

使用mongodb代理作为后端来获取Celery设置（遵循基本教程）。根据官方文档中规定的配置指南，我的celeryconfig.py设置如下：

CELERY_RESULT_BACKEND = "mongodb"
BROKER_BACKEND = "mongodb"
BROKER_URL = "mongodb://user:pass@subdomain.mongolab.com:123456/testdb"
CELERY_MONGODB_BACKEND_SETTINGS = {
    "host":"subdomain.mongolab.com",
    "port":123456,
    "database":"testdb",
    "taskmeta_collection":"taskmeta",
    "user":"user",
    "pass":"pass",
}
CELERY_IMPORTS = ("tasks",)

使用--loglevel=INFO运行celeryd会返回以下异常，该异常源自pymongo，但同时在kombu和celey中冒泡。

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/__init__.py", line 230, in start
    component.start()
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py", line 338, in start
    self.reset_connection()
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py", line 596, in reset_connection
    on_decode_error=self.on_decode_error)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/amqp.py", line 335, in get_task_consumer
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kombu/compat.py", line 187, in __init__
    super(ConsumerSet, self).__init__(self.backend, queues, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 285, in __init__
    self.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 295, in declare
    queue.declare()
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 388, in declare
    self.queue_declare(nowait, passive=False)
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 408, in queue_declare
    nowait=nowait)
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/virtual/__init__.py", line 380, in queue_declare
    return queue, self._size(queue), 0
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/mongodb.py", line 74, in _size
    return self.client.messages.find({"queue": queue}).count()
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/mongodb.py", line 171, in client
    self._client = self._open()
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/mongodb.py", line 97, in _open
    mongoconn = Connection(host=conninfo.hostname, port=conninfo.port)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/connection.py", line 325, in __init__
    nodes.update(uri_parser.split_hosts(entity, port))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/uri_parser.py", line 198, in split_hosts
    nodes.append(parse_host(entity, default_port))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/uri_parser.py", line 127, in parse_host
    raise ConfigurationError("Reserved characters such as ':' must be "
ConfigurationError: Reserved characters such as ':' must be escaped according RFC 2396. An IPv6 address literal must be enclosed in '[' and ']' according to RFC 2732.

Celery处理monguri的方式有些地方编码不正确，因为是pymongo中的uri解析器引发了这个错误。我尝试过转义uri字符串中的:字符，但所实现的只是使用损坏的连接字符串将传输重置回默认的AMQP。

amqp://guest@localhost:5672/mongodbhttp://user:password@subdomain.mongolab.com:29217/testdb

这显然是不对的。

我尝试使用r在配置中输入uri作为原始字符串，但没有任何更改。

我知道Celery从2.4开始就支持这种连接配置（我使用的是2.5.1，pymongo 2.1.1），官方文档都将其作为连接mongodb代理的首选方法。

这可能是一个错误，也许是与最新的pymongo版本不兼容？如果这种方法不起作用，如何将任务队列附加到副本集，因为我假设这些必须使用?replicaSet参数在monguri中传递。

我应该注意的是，我宁愿不使用RabbitMQ代理，因为Mongo已经在相关应用程序的堆栈中了，而且使用现有的应用程序似乎更直观。如果Mongo在这方面效果较差（每天的任务量相对较小）有什么具体原因的话，我很想知道！提前谢谢。

我认为这是一个错误。Celery将hostname而不是server_uri传递给kombu，因此导致了此问题。在跟踪代码后，我发现以下conf在他们修复错误之前绕过了它

CELERY_RESULT_BACKEND = 'mongodb'
BROKER_HOST = "subdomain.mongolab.com"
BROKER_PORT = 123456
BROKER_TRANSPORT = 'mongodb'
BROKER_VHOST = 'testdb'
CELERY_IMPORTS = ('tasks',)
CELERY_MONGODB_BACKEND_SETTINGS = { 
       'host': 'subdomain.mongolab.com',
       'port': 123456,
       'database': 'testdb',
       'user': user,
       'password': password,
       'taskmeta_collection': 'teskmeta'
       }

只是重复配置。

如果从CELERY_MONGODB_BACKEND_SETTINGS dict中删除"user"、"pass"、"port"one_answers"database"，并执行以下操作，会有帮助吗

BROKER_URL = "mongodb://user:pass@subdomain.mongolab.com:123456/testdb"
CELERY_MONGODB_BACKEND_SETTINGS = {
    "host":BROKER_URL,
    "taskmeta_collection":"taskmeta",
}

相关内容

最新更新

热门标签：