在弹性豆茎上与主管和Django一起运行芹菜

Question

我试图让芹菜与django一起在EB环境上运行，并且我已经非常接近使其运行。我有这个processor的配置文件

#!/usr/bin/env bash
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr 'n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=${celeryenv%?}
# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO
directory=/opt/python/current/app
user=ec2-user
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv,USER="ec2-user"
[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
  then
  echo "[include]" | tee -a /opt/python/etc/supervisord.conf
  echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
echo reread
supervisorctl -c /opt/python/etc/supervisord.conf reread
echo reload
# supervisorctl -c /opt/python/etc/supervisord.conf reload
# Update supervisord in cache without restarting all services
echo update
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
echo restart
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker

现在，当我部署IT时，芹菜节拍将运行(通过ssshing和ssshing in in and ssshing in and ps -aux |少于查看运行过程(，但没有芹菜工人。在我的活动中。我有这个。

[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Completed activity. Result:
  + chmod 0755 /var/run/httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --proxy httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --name application --location /opt/python/run/supervisord.pid
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
[2017-05-30T17:23:37.675Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
  [program:celeryd-worker]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO
  directory=/opt/python/current/app
  user=ec2-user
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10
  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600
  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true
  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998
  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"
  [program:celeryd-beat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django
  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-beat.log
  stderr_logfile=/var/log/celery-beat.log
  autostart=true
  autorestart=true
  startsecs=10
  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600
  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true
  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998
  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"
  reread
  No config updates to processes
  reload
  update
  restart
  celeryd-beat: stopped
  celeryd-beat: started
  celeryd-worker: ERROR (not running)
  celeryd-worker: ERROR (abnormal termination)

所以当我尝试启动工人而不是拍子时，有问题。

如果我进入EB并查看/var/log/celery-worker.log文件，我会看到此。

[2017-05-30 12:30:21,370: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 20, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 584, in start
    c.loop(*c.loop_args())
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/loops.py", line 88, in asynloop
    next(loop)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/hub.py", line 282, in create_loop
    item()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 139, in __call__
    return self.throw()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 136, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 290, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 306, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 315, in _get_async
    q, count=count, connection=self.asynsqs,
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 412, in asynsqs
    AsyncSQSConnection, _asynsqs.regions(),
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 400, in _aws_connect_to
    port=port)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/sqs/connection.py", line 39, in __init__
    https_connection_factory=https_connection_factory, **kwargs
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 223, in __init__
    AsyncConnection.__init__(self, http_client, **http_client_params)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 172, in __init__
    self._httpclient = http_client or get_client()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 22, in get_client
    client = hub._current_http_client = Client(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 13, in Client
    return CurlClient(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/curl.py", line 43, in __init__
    raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.

在本地机器上进行此操作时，我也遇到了这一点。尽管我确保使用此容器命令使用NSS安装卷曲。

container_commands:
  01_libcurl_ssl_backend:
    command: "export PYCURL_SSL_LIBRARY=nss"
    leader_only: true

Pycurl正在寻找的是。我的需求文件中也有pycurl，如果我冻结了冻结，我可以看到安装了pycurl。我通过重新创建我的虚拟环境在本地解决了这个问题。但不太确定该怎么做。

编辑1 我添加了

PYCURL_SSL_LIBRARY: nss

到我的配置文件中的选项设置，并且能够部署并显示出略有不同的行为。查看活动日志时。

而不是

update
restart
celeryd-beat: stopped
celeryd-beat: started
celeryd-worker: ERROR (not running)
celeryd-worker: ERROR (abnormal termination)

我得到了：

    reread
  celeryd-beat: changed
  celeryd-worker: changed
  reload
  update
  celeryd-beat: stopped
  celeryd-beat: updated process group
  celeryd-worker: stopped
  celeryd-worker: updated process group
  restart
  celeryd-beat: stopped
  celeryd-beat: started
  celeryd-worker: stopped
  celeryd-worker: ERROR (abnormal termination)

但是芹菜worker.log关于pycurl

Answer 1

1-遵循此问题的答案：https：//stackoverflow.com/a/41097851/3407986

2-您必须在包中添加libcurl -devel -eb配置文件/部分，请参阅我的。 ebextensions/02_packages.config file：

packages:
  yum:
    // ... your another packages where ...
    // add libcurl-devel, dependency of pycurl package
    libcurl-devel: []

Answer 2

尝试使用SSH到您的实例，然后使用EB使用的python来运行您的应用程序。然后您会看到实际错误。

例如，这是我在没有 --global-option="--with-nss"的pycurl构建pycurl时得到的：

[root@ip-172-31-2-149 log]# /opt/python/run/venv/local/bin/python Python 3.4.3 (default, Sep 1 2016, 23:33:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pycurl Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: pycurl: libcurl link-time ssl backend (nss) is different from compile-time ssl backend (none/other)