我已经安装了带有 2 个工人、1 个调度程序和 1 个网络服务器的 celeryexecutor 的 Airflow。我正在使用 systemd 启动所有这些组件。所有组件都启动良好，我可以在 Web UI 中看到 DAG。当我执行任何 DAG 时，第一个任务本身甚至没有启动它就失败了。在工人处启动任务似乎有些问题。以下是我在日志中得到的日志。
工作线程错误日志：
ct 21 19:17:23 mydomain.mycompany python[9780]: [2019-10-21 19:17:23,425: DEBUG/MainProcess] Task accepted: airflow.executors.celery_executor.execute_command[526ed99b-f21f-40ba-8018-afbf0dc9a36a] pid:9800
Oct 21 19:17:23 mydomain.mycompany python[9780]: [2019-10-21 19:17:23,425: INFO/Worker-16] Executing command in Celery: airflow run sip_data_parsing_batch_execution_dag clear_configurations 2019-10-21T19:17:20.802983+00:00 --local -sd /opt/sip-orchestration/airflow_pai/dags/sip_data_parsing/sip_data_parsing_dag.py
Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 2618, in _execute
Oct 21 19:17:24 mydomain.mycompany python[9780]: self.task_runner.start()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/bash_task_runner.py", line 34, in start
Oct 21 19:17:24 mydomain.mycompany python[9780]: self.process = self.run_command(['bash', '-c'], join_args=True)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/base_task_runner.py", line 126, in run_command
Oct 21 19:17:24 mydomain.mycompany python[9780]: preexec_fn=os.setsid
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
Oct 21 19:17:24 mydomain.mycompany python[9780]: restore_signals, start_new_session)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child
Oct 21 19:17:24 mydomain.mycompany python[9780]: raise child_exception_type(errno_num, err_msg, err_filename)
Oct 21 19:17:24 mydomain.mycompany python[9780]: FileNotFoundError: [Errno 2] No such file or directory: 'bash': 'bash'
Oct 21 19:17:24 mydomain.mycompany python[9780]: During handling of the above exception, another exception occurred:
Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/bin/airflow", line 32, in <module>
Oct 21 19:17:24 mydomain.mycompany python[9780]: args.func(args)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
Oct 21 19:17:24 mydomain.mycompany python[9780]: return f(*args, **kwargs)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/bin/cli.py", line 490, in run
Oct 21 19:17:24 mydomain.mycompany python[9780]: _run(args, dag, ti)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/bin/cli.py", line 401, in _run
Oct 21 19:17:24 mydomain.mycompany python[9780]: run_job.run()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 203, in run
Oct 21 19:17:24 mydomain.mycompany python[9780]: self._execute()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 2654, in _execute
Oct 21 19:17:24 mydomain.mycompany python[9780]: self.on_kill()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/jobs.py", line 2658, in on_kill
Oct 21 19:17:24 mydomain.mycompany python[9780]: self.task_runner.on_finish()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/bash_task_runner.py", line 44, in on_finish
Oct 21 19:17:24 mydomain.mycompany python[9780]: super(BashTaskRunner, self).on_finish()
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/task/task_runner/base_task_runner.py", line 163, in on_finish
Oct 21 19:17:24 mydomain.mycompany python[9780]: subprocess.call(['sudo', 'rm', self._cfg_path], close_fds=True)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 287, in call
Oct 21 19:17:24 mydomain.mycompany python[9780]: with Popen(*popenargs, **kwargs) as p:
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
Oct 21 19:17:24 mydomain.mycompany python[9780]: restore_signals, start_new_session)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child
Oct 21 19:17:24 mydomain.mycompany python[9780]: raise child_exception_type(errno_num, err_msg, err_filename)
Oct 21 19:17:24 mydomain.mycompany python[9780]: FileNotFoundError: [Errno 2] No such file or directory: 'sudo': 'sudo'
Oct 21 19:17:24 mydomain.mycompany python[9780]: [2019-10-21 19:17:24,681: ERROR/Worker-16] execute_command encountered a CalledProcessError
Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 58, in execute_command
Oct 21 19:17:24 mydomain.mycompany python[9780]: close_fds=True, env=env)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
Oct 21 19:17:24 mydomain.mycompany python[9780]: raise CalledProcessError(retcode, cmd)
Oct 21 19:17:24 mydomain.mycompany python[9780]: subprocess.CalledProcessError: Command 'airflow run sip_data_parsing_batch_execution_dag clear_configurations 2019-10-21T19:17:20.802983+00:00 --local -sd /opt/sip-orchestration/airflow_pai/dags/sip_data_parsing/sip_data_parsing_dag.py' returned non-zero exit status 1.
Oct 21 19:17:24 mydomain.mycompany python[9780]: [2019-10-21 19:17:24,682: ERROR/Worker-16] None
Oct 21 19:17:24 mydomain.mycompany python[9780]: [2019-10-21 19:17:24,684: ERROR/MainProcess] Task airflow.executors.celery_executor.execute_command[526ed99b-f21f-40ba-8018-afbf0dc9a36a] raised unexpected: AirflowException('Celery command failed',)
Oct 21 19:17:24 mydomain.mycompany python[9780]: Traceback (most recent call last):
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/celery/app/trace.py", line 240, in trace_task
Oct 21 19:17:24 mydomain.mycompany python[9780]: R = retval = fun(*args, **kwargs)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
Oct 21 19:17:24 mydomain.mycompany python[9780]: return self.run(*args, **kwargs)
Oct 21 19:17:24 mydomain.mycompany python[9780]: File "/opt/sip-orchestration/python/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 63, in execute_command
Oct 21 19:17:24 mydomain.mycompany python[9780]: raise AirflowException('Celery command failed')
Oct 21 19:17:24 mydomain.mycompany python[9780]: airflow.exceptions.AirflowException: Celery command failed
我用来启动组件的命令。
- 须藤服务气流工作线程启动
sudo 服务气流调度程序启动
须藤服务气流-网络服务器启动
气流版本 - 1.10.1 芹菜版本 - 3.1.17
当您的主机名不匹配时，也会发生此问题，因为您正在使用 Celery 执行器，您应该检查此链接。
您还应该检查为 Bash 运算符指定的路径或目录，您的工作人员也应该可以访问它。