我使用Python 3.6.5在codebuild容器中运行airflow 1.9.0，我们执行以下命令并获得错误
ValueError: Unable to configure handler 'file.processor':
'FileProcessorHandler' object has no attribute 'log'
sudo sh脚本/setup.sh
AIRFLOW_HOME="$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
export PIPENV_VENV_IN_PROJECT=True
cat <<EOF > $AIRFLOW_HOME/.env
PIPENV_VENV_IN_PROJECT=True
AIRFLOW_HOME=$AIRFLOW_HOME
EOF
cat <<EOF > $AIRFLOW_HOME/airflow.cfg
[core]
airflow_home = $AIRFLOW_HOME
dags_folder = $AIRFLOW_HOME/dags
base_log_folder = $AIRFLOW_HOME/logs
remote_log_conn_id =
encrypt_s3_logs = False
logging_level = INFO
logging_config_class =
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
executor = SequentialExecutor
sql_alchemy_conn = sqlite:///$AIRFLOW_HOME/airflow.db
sql_alchemy_pool_size = 5
sql_alchemy_pool_recycle = 3600
parallelism = 32
dag_concurrency = 16
dags_are_paused_at_creation = True
non_pooled_task_slot_count = 128
max_active_runs_per_dag = 16
load_examples = False
plugins_folder = $AIRFLOW_HOME/plugins
fernet_key = QfQ90TaCFkE9fSLJnoikJCRxj4cnqFX69D1_Lbnhinw=
donot_pickle = False
dagbag_import_timeout = 30
task_runner = BashTaskRunner
default_impersonation =
security =
unit_test_mode = False
task_log_reader = file.task
enable_xcom_pickling = True
killed_task_cleanup_time = 60
[cli]
api_client = airflow.api.client.local_client
endpoint_url = http://localhost:8080
[api]
auth_backend = airflow.api.auth.backend.default
[operators]
default_owner = Airflow
default_cpus = 1
default_ram = 512
default_disk = 512
default_gpus = 0
[webserver]
base_url = http://localhost:8080
web_server_host = 0.0.0.0
web_server_port = 8080
web_server_ssl_cert =
web_server_ssl_key =
web_server_worker_timeout = 120
worker_refresh_batch_size = 1
worker_refresh_interval = 30
secret_key = temporary_key
workers = 4
worker_class = sync
access_logfile = -
error_logfile = -
expose_config = False
authenticate = False
filter_by_owner = False
owner_mode = user
dag_default_view = tree
dag_orientation = LR
demo_mode = False
log_fetch_timeout_sec = 5
hide_paused_dags_by_default = False
page_size = 100
[email]
email_backend = airflow.utils.email.send_email_smtp
[dask]
cluster_address = 127.0.0.1:8786
[scheduler]
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5
run_duration = -1
min_file_process_interval = 0
dag_dir_list_interval = 300
print_stats_interval = 30
child_process_log_directory = $AIRFLOW_HOME/logs/scheduler
scheduler_zombie_task_threshold = 300
catchup_by_default = True
max_tis_per_query = 0
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
max_threads = 2
authenticate = False
[github_enterprise]
api_rev = v3
[admin]
hide_sensitive_variable_fields = True
EOF
pipenv --three install
. $(pwd)/.venv/bin/activate
我有这样的Pipfile，在其中我安装了所有的包来运行气流，它执行得很好
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
apache-airflow = {extras = ["celery", "crypto", "jdbc", "postgres", "s3",
"slack", "ssh", "redis", "log"], version = "==1.9.0"}
pymongo = "~=3.6.1"
boto3 = "~=1.7.40"
retrying = "~=1.3.3"
[dev-packages]
[requires]
python_version = "3.6"
在那之后，我有了我的buildspec.yml
- sudo sh scripts/setup.sh
- . $(pwd)/.venv/bin/activate
- airflow initdb
当我试图执行任何气流命令时，我都会遇到这个错误，我找不到任何东西来解决我有的问题
[Container] 2018/07/31 19:04:07 Running command airflow initdb
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/config.py", line 558, in configure
handler = self.configure_handler(handlers[name])
File "/usr/local/lib/python3.6/logging/config.py", line 731, in configure_handler
result = factory(**kwargs)
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
packages/airflow/utils/log/file_processor_handler.py", line 50, in __init__
self._symlink_latest_log_directory()
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site- packages/airflow/utils/log/file_processor_handler.py", line 110, in _symlink_latest_log_directory
self.log.warning(
AttributeError: 'FileProcessorHandler' object has no attribute 'log'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/codebuild/output/src571717869/src/.venv/bin/airflow", line 16, in
<module>
from airflow import configuration
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
packages/airflow/__init__.py", line 31, in <module>
from airflow import settings
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
packages/airflow/settings.py", line 148, in <module>
configure_logging()
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
packages/airflow/logging_config.py", line 75, in configure_logging
raise e
File "/codebuild/output/src571717869/src/.venv/lib/python3.6/site-
packages/airflow/logging_config.py", line 70, in configure_logging
dictConfig(logging_config)
File "/usr/local/lib/python3.6/logging/config.py", line 795, in dictConfig
dictConfigClass(config).configure()
File "/usr/local/lib/python3.6/logging/config.py", line 566, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'file.processor': 'FileProcessorHandler' object has no attribute 'log'
[Container] 2018/07/31 19:04:07 Command did not exit successfully airflow initdb exit status 1
我改变了执行Airflow的方式，它可以工作，还有一些关于配置的细节：
SO：Ubuntu 16.04Python 3.6.5
我从setup.sh中删除了以下几行，并将其放入buildspec.yml 中
pipenv --three install
. $(pwd)/.venv/bin/activate
对于我的执行，最好使用pipenv run而不是pipenv shell
(http://witkowskibartosz.com/blog/pipenv_run_vs_pipenv_shell.html#.W9e012hKjIU)
建筑规范yml
- bash scripts/setup.sh
- for var in $(cat .env) ; do export $var ; done
- pipenv --three install
- pipenv run airflow initdb
- pipenv run airflow list_dags
- pipenv run airflow list_tasks ${dag}
- pipenv run airflow backfill -s $(date -d "yesterday 13:00" '+%Y-%m-%d') ${dag}
- aws s3 cp --recursive --region us-east-1 $(pwd)/logs/ ${bucket_logs}