在码头工人中运行时面临来自 pydantic 的 kedro-气流验证错误



我是kedro和气流的新手。我正在尝试使用 docker 在气流中部署 kedro 管道。 但是在执行我的 DAG 时,我收到此错误:

2022-01-27 16:17:19,659 - airflow.task - ERROR - Task failed with exception
Traceback (most recent call last):
self._prepare_and_execute_task_with_callbacks(context, task)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1286, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1316, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/airflow/dags/adag.py", line 39, in execute
session.run(self.pipeline_name, node_names=[self.node_name])
File "/home/astro/.local/lib/python3.7/site-packages/kedro/framework/session/session.py", line 410, in run
run_params=record_data, pipeline=filtered_pipeline, catalog=catalog
File "/usr/local/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
return outcome.get_result()
File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/usr/local/lib/python3.7/site-packages/kedro_mlflow/framework/hooks/pipeline_hook.py", line 108, in before_pipeline_run
self._is_mlflow_enabled = _assert_mlflow_enabled(run_params["pipeline_name"])
File "/usr/local/lib/python3.7/site-packages/kedro_mlflow/framework/hooks/utils.py", line 8, in _assert_mlflow_enabled
mlflow_config = get_mlflow_config()
File "/usr/local/lib/python3.7/site-packages/kedro_mlflow/config/kedro_mlflow_config.py", line 228, in get_mlflow_config
mlflow_config = KedroMlflowConfig.parse_obj(conf_mlflow_yml)
File "pydantic/main.py", line 511, in pydantic.main.BaseModel.parse_obj
File "/usr/local/lib/python3.7/site-packages/kedro_mlflow/config/kedro_mlflow_config.py", line 106, in __init__
super().__init__(**kwargs)
File "pydantic/main.py", line 331, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for KedroMlflowConfig
hooks
extra fields not permitted (type=value_error.extra)

请帮助我理解和解决此问题。

免责声明:我是kedro-mlflow插件的作者,但我不是 kedro 核心团队的成员,我不知道kedro-airflow插件的内部结构

您的错误是由kedro-mlflow插件引发的,它表明您的mlflow.yml配置文件具有不支持的额外密钥。

使其再次工作的最简单方法是使用以下命令生成具有正确键的新配置文件:

kedro mlflow init --force

另一种方法是手动检查文件以找到您拥有的额外密钥。我怀疑您在创建项目时用kedro-mlflow==0.7.x生成了文件,并且您尝试在虚拟环境中运行该项目,kedro-mlflow==0.8.0几周前已经实现并且密钥略有不同。通常,我只在次要版本之间引入重大更改(一旦我达到 1.0.0,主要版本之间不会发生重大更改),因此您应该通过将需求与kedro-mlflow>=0.8.0, <0.9.0

最新更新