如何利用tensorflow夜间构建docker图像以及tensorflow目标检测研究模型

Question

由于tensorflow night对GPU的支持目前在Google Colab上已经中断，我正在尝试构建自己的docker镜像进行开发。但是，当我从tensorflow/models安装object_detection包时，我的夜间tensorflow包会被从object_detectionsetup.py作为依赖项引入的版本覆盖。

我在谷歌Colab中遵循了基本相同的步骤，但我每晚的tensorflow并没有被覆盖，所以我不确定我错过了什么。。。

这是我的Dockerfile:

FROM tensorflow/tensorflow:nightly-gpu-jupyter
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"
RUN apt-get install -y 
curl 
git 
less 
zip
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip
RUN cp bin/protoc /usr/local/bin
RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && 
protoc object_detection/protos/*.proto --python_out=. && 
cp object_detection/packages/tf2/setup.py . && 
python -m pip install .
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

我正在构建的：

docker pull tensorflow/tensorflow:nightly-gpu-jupyter
docker build --no-cache . -f models-tf-nightly.Dockerfile -t tf-nightly-models

第一个print()显示：

Tensorflow version: 2.5.0-dev20201129

但第二个显示：

Tensorflow version: 2.3.1

---

在Google Colab中，我正在执行基本相同的步骤：

# Install the Object Detection API
%%bash
pip install tf-nightly-gpu
[[ -d models ]] || git clone --depth 1 https://github.com/tensorflow/models
cd models/research/
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
python -m pip install .

之后

import tensorflow as tf
print(tf.__version__)

打印2.5.0-dev20201201

因此，不知何故，我的Google Colab步骤保留了我每晚的Tensorflow安装，而在Docker上，它会被2.3.0覆盖。

Answer 1

如果在安装对象检测包之前查看pip list，您会发现tf-nightly-gpu已安装，但tensorflow未安装。安装对象检测程序包时，tensorflow程序包将作为依赖项引入。pip认为它没有安装，所以安装了它。

解决这一问题的一种方法是欺骗pip-install，认为tensorflow包已经安装。可以通过符号链接dist-packages中的tf_nightly_gpu-VERSION.dist-info目录来实现这一点。我在下面的Dockerfile中添加了这样做的行。在这篇文章的底部，我还包含了一个Dockerfile，它实现了一些最小化图像大小的最佳实践。

FROM tensorflow/tensorflow:nightly-gpu-jupyter
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"
RUN apt-get install -y 
curl 
git 
less 
zip
# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-(.*)/1/')
WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip
RUN cp bin/protoc /usr/local/bin
RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && 
protoc object_detection/protos/*.proto --python_out=. && 
cp object_detection/packages/tf2/setup.py . && 
python -m pip install .
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

---

这是一个Dockerfile，它导致图像稍小(0.22 GB未压缩(。值得注意的变化是清除apt列表并在pip install中使用--no-cache-dir。

FROM tensorflow/tensorflow:nightly-gpu-jupyter
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"
RUN apt-get install -y --no-install-recommends 
ca-certificates 
curl 
git 
less 
zip && 
rm -rf /var/lib/apt/lists/*
# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-(.*)/1/')
WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && 
unzip protoc-3.11.4-linux-x86_64.zip && 
cp bin/protoc /usr/local/bin && 
rm -r protoc-3.11.4-linux-x86_64.zip bin/
# Upgrade pip.
RUN python -m pip install --no-cache-dir --upgrade pip
RUN git clone --depth 1 https://github.com/tensorflow/models
WORKDIR models/research
RUN protoc object_detection/protos/*.proto --python_out=. && 
cp object_detection/packages/tf2/setup.py . && 
python -m pip install  --no-cache-dir .
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"