无法通过请求 nvidia.com/gpu 来调度 Kubernetes Pod

Question

我已经能够让 Kubernetes 识别我节点上的 GPU：

$ kubectl get node MY_NODE -o yaml
...
allocatable:
  cpu: "48"
  ephemeral-storage: "15098429006"
  hugepages-1Gi: "0" 
  hugepages-2Mi: "0"
   memory: 263756344Ki
  nvidia.com/gpu: "8"
  pods: "110"
capacity:
  cpu: "48"
  ephemeral-storage: 16382844Ki
  hugepages-1Gi: "0"
  hugepages-2Mi: "0"
  memory: 263858744Ki
  nvidia.com/gpu: "8"
  pods: "110"
...

我用

Limits:
  cpu:             2
  memory:          2147483648
  nvidia.com/gpu:  1
Requests:
  cpu:             500m
  memory:          536870912
  nvidia.com/gpu:  1

但是，容器将保持挂起状态，并显示：

Insufficient nvidia.com/gpu.

我是否正确指定了资源？

Answer 1

您是否在 K8S 中安装了 NVIDIA 插件？

kubectl create -f nvidia.io/device-plugin.yml

某些设备太旧，无法进行健康检查。因此，必须禁用此选项：

containers:
      - image: nvidia/k8s-device-plugin:1.9
        name: nvidia-device-plugin-ctr
        env:
        - name: DP_DISABLE_HEALTHCHECKS
          value: "xids"

看看：

• 设备插件：https://kubernetes.io/docs/concepts/cluster-administration/device-plugins/
• NVIDIA github：https://github.com/NVIDIA/k8s-device-plugin