在哪里可以找到 Google Cloud Platform 深度学习 VM 映像和深度学习容器的源代码?



GCP 概述了深度学习 VM 中安装的内容,但查看实际的 shell 脚本可以更轻松地确定 VM 映像之间的确切差异、调试任何部署问题以及创建派生映像。

有人已经问过深度学习容器的Dockerfiles,但我想我会重复这个问题以增加它得到答案的几率。

您可以创建和设置本地深度学习容器。查看官方文档 本地深度学习容器入门,但请记住,此产品处于预发布状态,可能会更改或提供有限的支持。在这里,可以找到可供选择的容器映像。不幸的是,没有源代码。

同时,您可以按照 @John Hanley 的说明进行操作:"您可以拉取容器映像并显示容器的"历史记录"。这将向您显示它是如何构建的以及原始起始映像的内容。这并不意味着您可以访问原始图像。您可以添加和删除图像的某些部分。您还可以将图像导出到 tar 存档文件,修改然后重新导入"。

来自 @John Hanely 的说明为您提供了 docker 命令,以帮助了解容器是如何构建的。

如果输出中的第一个 docker 命令是 FROM,则可以在该基本容器上重复该过程。

$docker pull gcr.io/deeplearning-platform-release/tf-cpu:m96
...
$ docker history  gcr.io/deeplearning-platform-release/tf-cpu:m96 --format  " {{.CreatedBy}}" --no-trunc
RUN |2 VERSION=1-15 CONTAINER_NAME=tf-cpu/1-15 /bin/sh -c cd /opt/google/licenses &&     chmod +x query_licenses.sh &&     ./query_licenses.sh # buildkit
RUN |2 VERSION=1-15 CONTAINER_NAME=tf-cpu/1-15 /bin/sh -c BAZEL_INSTALLER_URL="https://github.com/bazelbuild/bazel/releases/download/0.19.0/bazel-0.19.0-installer-linux-x86_64.sh" &&     BAZEL_INSTALLER_FILE="bazel_installer.sh" &&     wget -q "${BAZEL_INSTALLER_URL}" -O "${BAZEL_INSTALLER_FILE}" &&     chmod +x "${BAZEL_INSTALLER_FILE}" &&     "./${BAZEL_INSTALLER_FILE}" &&     rm -rf "./${BAZEL_INSTALLER_FILE}" # buildkit
RUN |2 VERSION=1-15 CONTAINER_NAME=tf-cpu/1-15 /bin/sh -c export CONDA_REPOSITORY="/tmp/conda" &&     chmod +x /opt/google/conda/install_to_env.sh &&     ENV_DOCKER=1 /opt/google/conda/install_to_env.sh base dlenv-tf-${VERSION}-cpu-meta # buildkit
ENV KMP_SETTINGS=1
ENV KMP_AFFINITY=granularity=fine,verbose,compact,1,0
ENV KMP_BLOCKTIME=0
ENV CONTAINER_NAME=tf-cpu/1-15
ARG CONTAINER_NAME
LABEL com.google.environment=Container: TensorFlow 1-15
ARG VERSION
CMD ["/run_jupyter.sh"]
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c chmod +x run_jupyter.sh # buildkit
COPY build/container/run_jupyter.sh /run_jupyter.sh # buildkit
ENTRYPOINT ["/entrypoint.sh"]
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c chmod +x entrypoint.sh # buildkit
COPY build/container/entrypoint.sh /entrypoint.sh # buildkit
COPY build/package/conda/channels.json /opt/google/conda/channels.json # buildkit
COPY build/package/packages/jupyter/jupyter_notebook_config.py /opt/jupyter/.jupyter/jupyter_notebook_config.py # buildkit
COPY build/package/packages/jupyter/ipython_kernel_config.py /etc/ipython/ipython_kernel_config.py # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c chown -R "jupyter:jupyter" "/home/jupyter/." # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c adduser --uid 1000 --gid 1001 jupyter # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c addgroup --gid 1001 jupyter # buildkit
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/local/lib/x86_64-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:
VOLUME [/home/jupyter]
EXPOSE map[8080/tcp:{}]
ENV SHELL=/bin/bash
ENV PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c chmod +x /opt/google/conda/provision_conda.sh && /opt/google/conda/provision_conda.sh # buildkit
ENV DL_ANACONDA_HOME=/opt/conda
ENV ANACONDA_PYTHON_VERSION=3.7
COPY build/vm/packer/generic/packages /opt/google # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c cd / &&     gsutil cp -r gs://dl-platform-binaries-builds/openmpi-4.0.2/v20191105/openmpi.tar.gz  . &&     tar xf openmpi.tar.gz &&     rm -f openmpi.tar.gz # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c apt update -y &&     apt install -y libnuma-dev # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c if dpkg -s libnccl2; then         echo "deb https://packages.cloud.google.com/apt google-fast-socket main" | tee /etc/apt/sources.list.d/google-fast-socket.list &&         curl -s -L https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - &&         apt-get --allow-releaseinfo-change update && apt install -y google-fast-socket;     fi # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c apt-get --allow-releaseinfo-change update -y &&     apt-get install -y dirmngr &&     apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8B57C5C2836F4BEB &&     apt-key adv --keyserver keyserver.ubuntu.com --recv-keys FEEA9169307EA071 &&     apt-get --allow-releaseinfo-change update -y &&     echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list &&     curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - &&     apt-get --allow-releaseinfo-change update -y &&     apt-get install -y apt-transport-https ca-certificates gnupg &&     echo "deb http://packages.cloud.google.com/apt gcsfuse-focal main" | tee /etc/apt/sources.list.d/gcsfuse.list &&     curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - &&     apt-get --allow-releaseinfo-change update -y &&     apt-get install -y google-cloud-sdk && apt-get install -y gcsfuse &&     rm -rf /var/lib/apt/lists/* # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c apt-get --allow-releaseinfo-change -o Acquire::Check-Valid-Until=false update -y &&     apt-get install --no-install-recommends -y -q        $(grep -vE "^s*#" aptget-requirements.txt | tr "n" " ") &&     rm -rf /var/lib/apt/lists/* &&     rm -rf aptget-requirements.txt # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c if [ "${BASE_IMAGE}" =~ "^nvidia. *" ]; then       apt update -y || true && apt install -y wget && apt install -yq software-properties-common &&       wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin &&       mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 &&       apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub &&       add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" &&       apt-get --allow-releaseinfo-change update;     fi # buildkit
COPY build/vm/packer/base/aptget-requirements.txt /aptget-requirements.txt # buildkit
RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c apt update --allow-releaseinfo-change -y &&     apt upgrade -y # buildkit
ARG DEBIAN_FRONTEND=noninteractive
ENV LANG=C.UTF-8
ENV LC_ALL=C.UTF-8
LABEL com.google.environment=Container: Minimal
/bin/sh -c #(nop)  CMD ["bash"]
/bin/sh -c #(nop) ADD file:ff6963f777661fb16cc12fb04a97c558bd94768a6e4ab5bd90e01f3086818853 in /

最新更新