气流：值错误：无法配置处理程序"处理器" - wasb 记录器

Question

我正在尝试使用Azure Blob配置远程记录。

Airflow version: 1.10.2
Python: 3.6.5
Ubuntu: 18.04

以下是我所做的步骤：

1. 在$ airflow_home/config/log_config.py中，我已经放置了remote_base_log_folder ='wasb-airflow-logs'（这是容器内的文件夹（容器名称：airflow-logs））
）
2. 空 init .py在$ airflow_home/config/
3. $ airflow_home/config/添加在$ pythonpath中
4. 将default_logging_config重命名为$ airflow_home/config/log_config.py
到处登录config
5. 气流斑点连接中定义的用户已读/写入remote_base_log_folder
6. $ airflow_home/airflow.cfg它具有远程_logging = truelogging_config_class = log_config.logging_config远程_log_conn_id =

以下是错误：

Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 382, in resolve
    found = getattr(found, frag)
AttributeError: module 'airflow.utils.log' has no attribute 'wasb_task_handler'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 384, in resolve
    self.importer(used)
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/log/wasb_task_handler.py", line 23, in <module>
    from airflow.contrib.hooks.wasb_hook import WasbHook
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/contrib/hooks/wasb_hook.py", line 22, in <module>
    from airflow.hooks.base_hook import BaseHook
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 28, in <module>
    from airflow.models import Connection
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/models.py", line 86, in <module>
    from airflow.utils.dag_processing import list_py_file_paths
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/dag_processing.py", line 49, in <module>
    from airflow.settings import logging_class_path
ImportError: cannot import name 'logging_class_path'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 558, in configure
    handler = self.configure_handler(handlers[name])
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 708, in configure_handler
    klass = self.resolve(cname)
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 391, in resolve
    raise v
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 384, in resolve
    self.importer(used)
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/log/wasb_task_handler.py", line 23, in <module>
    from airflow.contrib.hooks.wasb_hook import WasbHook
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/contrib/hooks/wasb_hook.py", line 22, in <module>
    from airflow.hooks.base_hook import BaseHook
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 28, in <module>
    from airflow.models import Connection
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/models.py", line 86, in <module>
    from airflow.utils.dag_processing import list_py_file_paths
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/dag_processing.py", line 49, in <module>
    from airflow.settings import logging_class_path
ValueError: Cannot resolve 'airflow.utils.log.wasb_task_handler.WasbTaskHandler': cannot import name 'logging_class_path'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/gsingh/venv/bin/airflow", line 21, in <module>
    from airflow import configuration
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/__init__.py", line 36, in <module>
    from airflow import settings, configuration as conf
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/settings.py", line 262, in <module>
    logging_class_path = configure_logging()
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/logging_config.py", line 73, in configure_logging
    raise e
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
    dictConfig(logging_config)
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 795, in dictConfig
    dictConfigClass(config).configure()
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 566, in configure
    '%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': Cannot resolve 'airflow.utils.log.wasb_task_handler.WasbTaskHandler': cannot import name 'logging_class_path'

我不确定我缺少哪种配置。有人面临同样的问题吗？

Answer 1

i遇到相同的错误，但是如果我向上滚动滚动，我可以看到在Valueerror面前还有另一个例外。这是一个许可。

PermissionError: [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler'

我之所以遇到该错误的原因是因为我在运行气流Docker容器之前没有创建最初的3个文件夹（DAG，日志，插件）。因此，Docker似乎是自动创建的，但权限是错误的。

修复的步骤：

1. 停止电流容器

docker-compose down --volumes --remove-orphans

2. 删除文件夹dag，log，插件
3. 以防万一破坏已经创建的图像和卷（在Docker桌面中）
4. 再次从命令行创建文件夹

mkdir logs dags plugins

5. 再次运行气流码头

docker-compose up airflow-init
docker-compose up

Answer 2

您需要安装Azure软件包。

pip install 'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]

按更新。md

现在应该使用

安装此

pip install apache-airflow[azure]

但这对我不起作用。

Answer 3

sudo chown 50000:0 dags logs plugins在我的情况下。

我试图用所有这些容器（取决于这3卷向前）运行官方的docker-compose.yml，或者只是将airflow standalone包装到一个容器中以进行调试目的。事实证明，创建了用根所有权而不是气流创建的。