AWS MWAA发布以运行DBT

Question

我在这里学习AWS关于使用MWAA运行DBT的教程。我在s3中复制了dbt-start项目(我的bucket/dags/dbt/dbt-starter项目(，并将教程中的两个DAG添加到了我的bucket/dags文件夹中。

第一个检查安装是否正确的代码是：

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.utils.dates import days_ago
with DAG(dag_id="dbt-installation-test", schedule_interval=None, catchup=False, start_date=days_ago(1)) as dag:
cli_command = BashOperator(
task_id="bash_command",
bash_command="/usr/local/airflow/.local/bin/dbt --version"
)

但是DAG在中失败了

[2022-10-01, 10:10:38 UTC] {{taskinstance.py:1262}} INFO - Executing <Task(BashOperator): bash_command> on 2022-10-01 10:10:37.699795+00:00
[2022-10-01, 10:10:38 UTC] {{standard_task_runner.py:52}} INFO - Started process 515 to run task
[2022-10-01, 10:10:38 UTC] {{standard_task_runner.py:76}} INFO - Running: ['airflow', 'tasks', 'run', 'dbt-installation-test', 'bash_command', 'manual__2022-10-01T10:10:37.699795+00:00', '--job-id', '20', '--raw', '--subdir', 'DAGS_FOLDER/dag_check_dbt.py', '--cfg-path', '/tmp/tmpw5qjhl4p', '--error-file', '/tmp/tmpanvrgrxj']
[2022-10-01, 10:10:38 UTC] {{standard_task_runner.py:77}} INFO - Job 20: Subtask bash_command
[2022-10-01, 10:10:38 UTC] {{logging_mixin.py:109}} INFO - Running <TaskInstance: dbt-installation-test.bash_command manual__2022-10-01T10:10:37.699795+00:00 [running]> on host ip-172-27-4-81.eu-west-1.compute.internal
[2022-10-01, 10:10:38 UTC] {{taskinstance.py:1429}} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=dbt-installation-test
AIRFLOW_CTX_TASK_ID=bash_command
AIRFLOW_CTX_EXECUTION_DATE=2022-10-01T10:10:37.699795+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-10-01T10:10:37.699795+00:00
[2022-10-01, 10:10:38 UTC] {{subprocess.py:62}} INFO - Tmp dir root location: 
/tmp
[2022-10-01, 10:10:38 UTC] {{subprocess.py:74}} INFO - Running command: ['bash', '-c', '/usr/local/airflow/.local/bin/dbt --version']
[2022-10-01, 10:10:38 UTC] {{subprocess.py:85}} INFO - Output:
[2022-10-01, 10:10:38 UTC] {{subprocess.py:89}} INFO - bash: /usr/local/airflow/.local/bin/dbt: No such file or directory
[2022-10-01, 10:10:38 UTC] {{subprocess.py:93}} INFO - Command exited with return code 127
[2022-10-01, 10:10:38 UTC] {{taskinstance.py:1703}} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
result = execute_callable(context=context)
File "/usr/local/lib/python3.7/site-packages/airflow/operators/bash.py", line 188, in execute
f'Bash command failed. The command returned a non-zero exit code {result.exit_code}.'
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 127.
[2022-10-01, 10:10:38 UTC] {{taskinstance.py:1280}} INFO - Marking task as FAILED. dag_id=dbt-installation-test, task_id=bash_command, execution_date=20221001T101037, start_date=20221001T101038, end_date=20221001T101038
[2022-10-01, 10:10:39 UTC] {{standard_task_runner.py:91}} ERROR - Failed to execute job 20 for task bash_command
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
args.func(args, dag=self.dag)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
_run_raw_task(args, ti)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184, in _run_raw_task
error_file=args.error_file,
File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
result = execute_callable(context=context)
File "/usr/local/lib/python3.7/site-packages/airflow/operators/bash.py", line 188, in execute
f'Bash command failed. The command returned a non-zero exit code {result.exit_code}.'
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 127.
[2022-10-01, 10:10:39 UTC] {{local_task_job.py:154}} INFO - Task exited with return code 1
[2022-10-01, 10:10:39 UTC] {{local_task_job.py:264}} INFO - 0 downstream tasks scheduled from follow-on schedule check

这是否意味着安装不成功？如果测试失败，AWS没有提供任何故障排除。

requirements.txt的内容：

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt"
apache-airflow[postgres,mysql,google]==2.2.2
SQLAlchemy==1.3.24
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
sentry-sdk==1.4.3
google-ads
tableauserverclient
facebook-business
openpyxl
airflow-provider-great-expectations==0.1.1
XlsxWriter
json-rpc==1.13.0
minimal-snowplow-tracker==0.0.2
packaging==20.9
networkx==2.6.3 
mashumaro==2.5
sqlparse==0.4.2
logbook==1.5.3
agate==1.6.1
dbt-extractor==0.4.0
pyparsing==2.4.7 
msgpack==1.0.2
parsedatetime==2.6
pytimeparse==1.1.8
leather==0.3.4
pyyaml==5.4.1
# Airflow constraints are jsonschema==3.2.0
jsonschema==3.1.1
hologram==0.0.14
dbt-core==0.21.1
psycopg2-binary==2.8.6
dbt-postgres==0.21.1
dbt-redshift==0.21.1

Answer 1

您的安装路径似乎略有不同。您正在尝试在/usr/local/airflow/.local/bin/dbt中访问dbt，这意味着Python环境将是/usr/local/airflow/.local。

然而，您的气流从/usr/local/lib/python3.7/site-packages/airflow运行，这意味着Python环境将安装在/usr/local/中。

尝试将bash命令更改为：

/usr/local/bin/dbt --version