我在COS上使用了T4 GPU的计算引擎VM很长一段时间，直到最近cos-extensions install gpu不像以前那样工作。
I0830 07:32:58.419130 987 main.go:21] Checking if this is the only cos_gpu_installer that is running.
I0830 07:32:58.427417 987 install.go:74] Running on COS build id 16108.470.16
I0830 07:32:58.427566 987 installer.go:187] Getting the default GPU driver version
I0830 07:32:58.427911 987 utils.go:72] Downloading gpu_default_version from https://storage.googleapis.com/cos-tools/16108.470.16/gpu_default_version
I0830 07:32:58.548403 987 utils.go:120] Successfully downloaded gpu_default_version from https://storage.googleapis.com/cos-tools/16108.470.16/gpu_default_version
I0830 07:32:58.548594 987 install.go:85] Installing GPU driver version 450.119.04
I0830 07:32:58.549646 987 cache.go:72] map[BUILD_ID:16108.470.11 DRIVER_VERSION:450.119.04]
I0830 07:32:58.549674 987 install.go:120] Did not find cached version, installing the drivers...
I0830 07:32:58.549681 987 installer.go:82] Configuring driver installation directories
I0830 07:32:58.563327 987 installer.go:196] Updating container's ld cache
I0830 07:32:58.793692 987 signature.go:30] Downloading driver signature for version 450.119.04
I0830 07:32:58.793721 987 utils.go:72] Downloading 450.119.04.signature.tar.gz from https://storage.googleapis.com/cos-tools/16108.470.16/extensions/gpu/450.119.04.signature.tar.gz
E0830 07:32:58.828902 987 artifacts.go:106] Failed to download extensions/gpu/450.119.04.signature.tar.gz from public GCS: failed to download 450.119.04.signature.tar.gz, status: 404 Not Found
E0830 07:32:58.829401 987 install.go:175] failed to download driver signature: failed to download driver signature for version 450.119.04: failed to download extensions/gpu/450.119.04.signature.tar.gz
安装程序似乎找不到驱动程序签名。我已经研究了这个问题，并通过执行
来遵循解决方法。/usr/bin/docker run --rm
--privileged
--net=host
--pid=host
--volume /dev:/dev
--volume /:/root
--volume /var/lib/toolbox/nvidia:/usr/local/nvidia
--env NVIDIA_DRIVER_VERSION=450.119.04
gcr.io/cos-cloud/cos-gpu-installer:latest
却得到了这个
+ COS_KERNEL_INFO_FILENAME=kernel_info
+ COS_KERNEL_SRC_HEADER=kernel-headers.tgz
+ TOOLCHAIN_URL_FILENAME=toolchain_url
+ TOOLCHAIN_ENV_FILENAME=toolchain_env
+ TOOLCHAIN_PKG_DIR=/build/cos-tools
+ CHROMIUMOS_SDK_GCS=https://storage.googleapis.com/chromiumos-sdk
+ ROOT_OS_RELEASE=/root/etc/os-release
+ KERNEL_SRC_HEADER=/build/usr/src/linux
+ NVIDIA_DRIVER_VERSION=450.119.04
+ NVIDIA_DRIVER_MD5SUM=
+ NVIDIA_INSTALL_DIR_HOST=/var/lib/nvidia
+ NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia
+ ROOT_MOUNT_DIR=/root
+ CACHE_FILE=/usr/local/nvidia/.cache
+ LOCK_FILE=/root/tmp/cos_gpu_installer_lock
+ LOCK_FILE_FD=20
+ set +x
[INFO 2021-08-30 07:36:38 UTC] PRELOAD: false
[INFO 2021-08-30 07:36:38 UTC] Running on COS build id 16108.470.16
[INFO 2021-08-30 07:36:38 UTC] Data dependencies (e.g. kernel source) will be fetched from https://storage.googleapis.com/cos-tools/16108.470.16
[INFO 2021-08-30 07:36:38 UTC] Checking if this is the only cos-gpu-installer that is running.
[INFO 2021-08-30 07:36:38 UTC] Checking if third party kernel modules can be installed
/tmp/esp /
/
[INFO 2021-08-30 07:36:38 UTC] Checking cached version
/entrypoint.sh: line 172: CACHE_BUILD_ID: unbound variable
似乎COS和COS GPU驱动程序发生了一些变化(可能?)，但只是想知道除了等待GCP解决问题外，是否有解决这个问题的方法。
这与Jan Vansteenlandt链接的案例相同。
这在某些版本的COS中发生;
例如最新的稳定COS版本- 89-16108:
vm-16108 ~ # cos-extensions list Available extensions for COS version
89-16108.470.16:
[gpu]
在[gpu]下没有列出驱动程序，并且运行cos-extensions install gpu以与您的情况相同的方式结束。当尝试运行您提到的docker容器时，也产生了相同的结果。
这是一个已知的问题，已经在IssueTracker上提出了． 你可以点击+1按钮，也可以在帖子中评论和发布你自己的发现。
在线程中也有一个解决方案，所以你可以给它一个尝试。
如果你可以使用旧版本的COS(例如85-13310)-驱动程序列出:
vm-13310 ~ # cos-extensions list
Available extensions for COS version 85-13310.1308.10:
[gpu]
450.119.04 [default]
当你运行cos-extensions install gpu时，它将导致NVIDIA驱动程序的成功安装:
vm-13310 ~ # cos-extensions install gpu
I0831 14:25:11.405591 1168 main.go:21] Checking if this is the only cos_gpu_installer that is running.
I0831 14:25:11.407510 1168 install.go:74] Running on COS build id 13310.1308.10
I0831 14:25:11.407519 1168 installer.go:187] Getting the default GPU driver version
I0831 14:25:11.407581 1168 utils.go:72] Downloading gpu_default_version from https://storage.googleapis.com/cos-tools/13310.1308.10/gpu_default_version
I0831 14:25:11.448046 1168 utils.go:120] Successfully downloaded gpu_default_version from https://storage.googleapis.com/cos-tools/13310.1308.10/gpu_default_version
I0831 14:25:11.448539 1168 install.go:85] Installing GPU driver version 450.119.04
I0831 14:25:11.448751 1168 cache.go:69] error: failed to read file /root/var/lib/nvidia/.cache: open /root/var/lib/nvidia/.cache: no such file or directory
I0831 14:25:11.448942 1168 install.go:120] Did not find cached version, installing the drivers...
I0831 14:25:11.449084 1168 installer.go:82] Configuring driver installation directories
I0831 14:25:11.469718 1168 installer.go:196] Updating container's ld cache
I0831 14:25:11.480682 1168 signature.go:30] Downloading driver signature for version 450.119.04
I0831 14:25:11.481007 1168 utils.go:72] Downloading 450.119.04.signature.tar.gz from https://storage.googleapis.com/cos-tools/13310.1308.10/extensions/gpu/450.119.04.signature.tar.gz
I0831 14:25:11.506186 1168 utils.go:120] Successfully downloaded 450.119.04.signature.tar.gz from https://storage.googleapis.com/cos-tools/13310.1308.10/extensions/gpu/450.119.04.signature.tar.gz
I0831 14:25:11.506541 1168 signature.go:37] Decompressing signature /build/sign-gpu-driver/450.119.04.signature.tar.gz
I0831 14:25:11.510104 1168 installer.go:68] Downloading GPU driver installer version 450.119.04
I0831 14:25:11.511637 1168 utils.go:72] Downloading GPU driver installer from https://storage.googleapis.com/nvidia-drivers-us-public/nvidia-cos-project/85/tesla/450_00/450.119.04/NVIDIA-Linux-x86_64-450.119.04_85-13310-1308-10.cos
I0831 14:25:12.885856 1168 utils.go:120] Successfully downloaded GPU driver installer from https://storage.googleapis.com/nvidia-drivers-us-public/nvidia-cos-project/85/tesla/450_00/450.119.04/NVIDIA-Linux-x86_64-450.119.04_85-13310-1308-10.cos
----- removed some lines for better readibility -----
I0831 14:28:49.433597 1168 cache.go:58] Updated cached version as
I0831 14:28:49.498379 1168 cache.go:60] BUILD_ID=13310.1308.10
I0831 14:28:49.498560 1168 cache.go:60] DRIVER_VERSION=450.119.04
I0831 14:28:49.498694 1168 installer.go:32] Verifying GPU driver installation
I0831 14:28:50.309502 1168 utils.go:334] Tue Aug 31 14:28:50 2021
I0831 14:28:50.309879 1168 utils.go:334] +-----------------------------------------------------------------------------+
I0831 14:28:50.311093 1168 utils.go:334] | NVIDIA-SMI 450.119.04 Driver Version: 450.119.04 CUDA Version: 11.0 |
I0831 14:28:50.311300 1168 utils.go:334] |-------------------------------+----------------------+----------------------+
I0831 14:28:50.311497 1168 utils.go:334] | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
I0831 14:28:50.311640 1168 utils.go:334] | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
I0831 14:28:50.311784 1168 utils.go:334] | | | MIG M. |
I0831 14:28:50.311949 1168 utils.go:334] |===============================+======================+======================|
I0831 14:28:50.322257 1168 utils.go:334] | 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
I0831 14:28:50.322566 1168 utils.go:334] | N/A 76C P0 27W / 70W | 0MiB / 15109MiB | 0% Default |
I0831 14:28:50.322708 1168 utils.go:334] | | | N/A |
I0831 14:28:50.322878 1168 utils.go:334] +-------------------------------+----------------------+----------------------+
I0831 14:28:50.323119 1168 utils.go:334]
I0831 14:28:50.323293 1168 utils.go:334] +-----------------------------------------------------------------------------+
I0831 14:28:50.323431 1168 utils.go:334] | Processes: |
I0831 14:28:50.323597 1168 utils.go:334] | GPU GI CI PID Type Process name GPU Memory |
I0831 14:28:50.323715 1168 utils.go:334] | ID ID Usage |
I0831 14:28:50.323863 1168 utils.go:334] |=============================================================================|
I0831 14:28:50.324222 1168 utils.go:334] | No running processes found |
I0831 14:28:50.324439 1168 utils.go:334] +-----------------------------------------------------------------------------+
I0831 14:28:50.465730 1168 modules.go:48] Updating host's ld cache
I0831 14:28:52.305122 1168 install.go:167] Finished installing the drivers.