我正在尝试运行一个DAG,它有一个运行jupyter笔记本的任务。每项任务都做得很好,直到交给造纸厂操作员。进度条停留在0上,任务永远不会完成。当观察吊舱内部时,有一个:
AirflowException : "Celery command failed on host <airflow-worker id>"
i'm using airflow 2.3.3 and the pod have papermill(2.4.0) and ipykernel(6.15.2) installed
Arflow is running on openshift
解决方案是为pod提供更多的资源(cpu和内存(。
在你的dagger里:
from kubernetes.client import models as k8s
task_ressource = k8s.V1ResourceRequirements(limits={"cpu": "800m", "memory": "5Gi"}, request={"cpu": "200m", "memory": "2Gi"})
task = KubernetesPodOperator('task_id'='task_id',image='your image',resources=task_ressource)