CeleryD似乎忽略了并发性的争论

Question

我最近将Django项目升级到了Celery 4.4.6，但进展并不顺利。我目前的首要问题是任务的并发性。因为这些任务锁定数据库表，而且有些任务占用大量内存，所以不可能同时运行八个任务。我也只有一台2处理器的机器可用。然而，这正是芹菜所要做的。

以前我只能同时执行两项任务。

工作进程处于守护状态，并且只有一个工作进程处于活动状态(一个节点(。我将并发性设置为2。这是我的/etc/default/celeryd:

#   most people will only start one node:
CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
#   alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=3
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/ubuntu/dev/bin/python -m celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"
# App instance to use
# comment out this line if you don't use an app
CELERY_APP="match2"
# or fully qualified:
#CELERY_APP="proj.tasks:app"
# Where to chdir at start.
export DJANGO_SETTINGS_MODULE="match2.settings"
CELERYD_CHDIR="/home/ubuntu/dev/match2/match2"
# Extra command-line arguments to the worker
CELERYD_OPTS="--concurrency=2"
# Configure node-specific settings by appending node name to arguments:
#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"
# Set logging level to DEBUG
CELERYD_LOG_LEVEL="INFO"
# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists (e.g., nobody).
CELERYD_USER="ubuntu"
CELERYD_GROUP="users"
# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

我在很大程度上假设这一行将负责同时执行多少任务：CELERYD_OPTS="--concurrency=2"但它似乎仍然从RabbitMQ消息队列中提取了多达八个项目。

感谢您的帮助。

Answer 1

因此，经过一番反复和加入谷歌集团，我终于得到了答案：

如果你想让Celery表现得像一个优秀的小工人，并且在完成旧任务之前不承担另一项任务，你需要在你的设置文件中同时拥有这两项：

task_acks_late = True
worker_prefetch_multiplier = 1

如果您在Django项目中使用它，使用老式的大写设置(请参阅：https://docs.celeryproject.org/en/stable/userguide/configuration.html#new-小写设置(，则将其转换为：

CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_TASK_ACKS_LATE = True