气流被卡住,运行PapermillOperator永远不会结束



我正在尝试运行一个DAG,它有一个运行jupyter笔记本的任务。每项任务都做得很好,直到交给造纸厂操作员。进度条停留在0上,任务永远不会完成。当观察吊舱内部时,有一个:

AirflowException : "Celery command failed on host <airflow-worker id>"
i'm using airflow 2.3.3 and the pod have papermill(2.4.0) and ipykernel(6.15.2) installed
Arflow is running on openshift

解决方案是为pod提供更多的资源(cpu和内存(。

在你的dagger里:

from kubernetes.client import models as k8s
task_ressource = k8s.V1ResourceRequirements(limits={"cpu": "800m", "memory": "5Gi"}, request={"cpu": "200m", "memory": "2Gi"})
task = KubernetesPodOperator('task_id'='task_id',image='your image',resources=task_ressource)

最新更新