docker语言 - -cache-from with BUILDKIT_INLINE_CACHE 不是每隔一段时间都工作

Question

我正在尝试利用 BUILDKIT for Docker 的缓存/拉取系统来执行我的 CI/CD 过程。但它没有按预期工作.
我创建了一个虚拟的本地示例(但在我的 CI 系统 - AWS CodePipeline 以及 DockerHub 和 AWS ECR 中也会发生同样的情况(。 Dockerfile：

# base image
FROM python:3.7-slim
# set working directory
WORKDIR /usr/src/app
# add and install requirements
RUN pip install --upgrade pip
COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip $PIP_PROXY install --no-cache-dir --compile -r requirements.txt
RUN echo 123
# add app
COPY ./run_test.py /usr/src/app/run_test.py
# run server
CMD ["python", "run_test.py"]

run_test.py实际上并不有趣，但这是以防万一的代码：

import requests
import time
while True:
time.sleep(1)
print(requests)

您还需要在同一文件夹中创建一个空的requirements.txt文件.
事先，我导出了两个环境变量：

export DOCKER_BUILDKIT=1  # to activate buildkit
export DUMMY_IMAGE_URL=bi0max/test_docker

然后，为了测试我有以下命令。前两个命令删除本地缓存以类似于 CI 环境，然后构建和推送.
请注意，下面的代码会删除本地构建缓存：

docker builder prune -a -f && 
(docker image rm $DUMMY_IMAGE_URL:latest || true) && 
docker build 
--cache-from $DUMMY_IMAGE_URL:latest 
--build-arg BUILDKIT_INLINE_CACHE=1 
--tag $DUMMY_IMAGE_URL:latest "." && 
docker push $DUMMY_IMAGE_URL:latest

正如预期的那样，第一次运行只是从头开始构建所有内容：

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 434B done
#2 DONE 0.0s
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.1s
#3 [internal] load metadata for docker.io/library/python:3.7-slim
#3 DONE 0.0s
#12 [1/7] FROM docker.io/library/python:3.7-slim
#12 DONE 0.0s
#7 [internal] load build context
#7 DONE 0.0s
#4 importing cache manifest from bi0max/test_docker:latest
#4 ERROR: docker.io/bi0max/test_docker:latest not found
#12 [1/7] FROM docker.io/library/python:3.7-slim
#12 resolve docker.io/library/python:3.7-slim done
#12 DONE 0.0s
#7 [internal] load build context
#7 transferring context: 204B done
#7 DONE 0.1s
#5 [2/7] WORKDIR /usr/src/app
#5 DONE 0.0s
#6 [3/7] RUN pip install --upgrade pip
#6 1.951 Requirement already up-to-date: pip in /usr/local/lib/python3.7/site-packages (20.1.1)
#6 DONE 2.3s
#8 [4/7] COPY ./requirements.txt /usr/src/app/requirements.txt
#8 DONE 0.0s
#9 [5/7] RUN pip $PIP_PROXY install --no-cache-dir --compile -r requirement...
#9 0.750 Collecting requests==2.22.0
#9 0.848   Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
#9 0.932 Collecting idna<2.9,>=2.5
#9 0.948   Downloading idna-2.8-py2.py3-none-any.whl (58 kB)
#9 0.995 Collecting chardet<3.1.0,>=3.0.2
#9 1.011   Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
#9 1.135 Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
#9 1.153   Downloading urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
#9 1.264 Collecting certifi>=2017.4.17
#9 1.282   Downloading certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
#9 1.378 Installing collected packages: idna, chardet, urllib3, certifi, requests
#9 1.916 Successfully installed certifi-2020.4.5.1 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.9
#9 DONE 2.2s
#10 [6/7] RUN echo 123
#10 0.265 123
#10 DONE 0.3s
#11 [7/7] COPY ./run_test.py /usr/src/app/run_test.py
#11 DONE 0.0s
#13 exporting to image
#13 exporting layers done
#13 writing image sha256:f98327afae246096725f7e54742fe9b25079f1b779699b099e66c8def1e19052 done
#13 naming to docker.io/bi0max/test_docker:latest done
#13 DONE 0.0s
#14 exporting cache
#14 preparing build cache for export done
#14 DONE 0.0s

然后，我稍微调整run_test.py文件，结果再次如预期的那样。在最后一步([7/7] COPY(之前的所有图层都从存储库下载并重复使用。

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 434B done
#1 DONE 0.1s
#3 [internal] load metadata for docker.io/library/python:3.7-slim
#3 DONE 0.0s
#8 [internal] load build context
#8 DONE 0.0s
#4 [1/7] FROM docker.io/library/python:3.7-slim
#4 DONE 0.0s
#5 importing cache manifest from bi0max/test_docker:latest
#5 DONE 1.2s
#8 [internal] load build context
#8 transferring context: 193B done
#8 DONE 0.0s
#6 [2/7] WORKDIR /usr/src/app
#6 CACHED
#7 [3/7] RUN pip install --upgrade pip
#7 CACHED
#9 [4/7] COPY ./requirements.txt /usr/src/app/requirements.txt
#9 CACHED
#10 [5/7] RUN pip $PIP_PROXY install --no-cache-dir --compile -r requirement...
#10 CACHED
#11 [6/7] RUN echo 123
#11 pulling sha256:79fc69c08b391d082b4d2617faed489d220444fa0cf06953cdff55c667866bed
#11 pulling sha256:071624272167ab4e35a30eb1640cb3f15ced19c6cd10fa1c9d49763372e81c23
#11 pulling sha256:04ed4ecd76e1a110f468eb1a3173bbfa578c6b4c85a6dc82bf4a489ed8b8c54d
#11 pulling sha256:79fc69c08b391d082b4d2617faed489d220444fa0cf06953cdff55c667866bed 0.2s done
#11 pulling sha256:d6406c1ce2dc5e841233ebce164ee469388102cb98f1473adaeca15455d6d797
#11 pulling sha256:071624272167ab4e35a30eb1640cb3f15ced19c6cd10fa1c9d49763372e81c23 0.5s done
#11 pulling sha256:04ed4ecd76e1a110f468eb1a3173bbfa578c6b4c85a6dc82bf4a489ed8b8c54d 0.5s done
#11 pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#11 pulling sha256:d6406c1ce2dc5e841233ebce164ee469388102cb98f1473adaeca15455d6d797 0.3s done
#11 pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.2s done
#11 CACHED
#12 [7/7] COPY ./run_test.py /usr/src/app/run_test.py
#12 DONE 0.0s
#13 exporting to image
#13 exporting layers done
#13 writing image sha256:f37692114f10b9a3646203569a0849af20774651f4aa0f5dc8d6f133fb7ff062 done
#13 naming to docker.io/bi0max/test_docker:latest done
#13 DONE 0.0s
#14 exporting cache
#14 preparing build cache for export done
#14 DONE 0.0s

现在，我再次更改run_test.py，我希望 docker 能做与上次相同的事情。但是我得到了以下结果，它从头开始构建所有内容：

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 434B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.7-slim
#3 DONE 0.0s
#5 [1/7] FROM docker.io/library/python:3.7-slim
#5 DONE 0.0s
#8 [internal] load build context
#8 DONE 0.0s
#4 importing cache manifest from bi0max/test_docker:latest
#4 DONE 1.7s
#8 [internal] load build context
#8 transferring context: 182B done
#8 DONE 0.0s
#5 [1/7] FROM docker.io/library/python:3.7-slim
#5 resolve docker.io/library/python:3.7-slim done
#5 DONE 0.1s
#6 [2/7] WORKDIR /usr/src/app
#6 DONE 0.0s
#7 [3/7] RUN pip install --upgrade pip
#7 1.774 Requirement already up-to-date: pip in /usr/local/lib/python3.7/site-packages (20.1.1)
#7 DONE 2.1s
#9 [4/7] COPY ./requirements.txt /usr/src/app/requirements.txt
#9 DONE 0.0s
#10 [5/7] RUN pip $PIP_PROXY install --no-cache-dir --compile -r requirement...
#10 0.805 Collecting requests==2.22.0
#10 0.905   Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
#10 1.079 Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
#10 1.109   Downloading urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
#10 1.242 Collecting certifi>=2017.4.17
#10 1.259   Downloading certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
#10 1.336 Collecting idna<2.9,>=2.5
#10 1.353   Downloading idna-2.8-py2.py3-none-any.whl (58 kB)
#10 1.410 Collecting chardet<3.1.0,>=3.0.2
#10 1.428   Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
#10 1.545 Installing collected packages: urllib3, certifi, idna, chardet, requests
#10 2.102 Successfully installed certifi-2020.4.5.1 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.9
#10 DONE 2.4s
#11 [6/7] RUN echo 123
#11 0.259 123
#11 DONE 0.3s
#12 [7/7] COPY ./run_test.py /usr/src/app/run_test.py
#12 DONE 0.0s
#13 exporting to image
#13 exporting layers done
#13 writing image sha256:f4ffb0e84e334b4b35fe2504de11012e5dc1ca5978eace055932e9bbbe83c93e done
#13 naming to docker.io/bi0max/test_docker:latest done
#13 DONE 0.0s
#14 exporting cache
#14 preparing build cache for export done
#14 DONE 0.0s

但对我来说最奇怪的是，当我第三次更改run_test.py时，它再次使用缓存层。它以同样的方式继续：第四次 - 不使用，第五次 - 使用，等等......

我在这里错过了什么吗？

如果我每次在构建之前拉取映像，那么它总是使用缓存，但它在没有 BUILDKIT 的情况下也以相同的方式工作。

Answer 1

此问题已在较新的 docker 版本中修复，只需升级即可解决此问题。

否则，GitHub 上描述的解决方案可以帮助不依赖于系统 docker 版本：https://github.com/moby/buildkit/issues/1981#issuecomment-785534131

Answer 2

我相信如果在重用缓存时构建内联缓存图像，它将变得无效(或不完整(。这要么是限制，要么是错误。

有一个解决方法：可以标记不同的缓存映像，只有在 BuildKit 重新生成映像时才会推送到注册表。AFAIK 没有办法知道 BuildKit 是否使用了缓存，但我们可以看到日志在运行时被 CACHED 填充，因此我们可以重用它。例如：

# enable buildkit:
$ export DOCKER_BUILDKIT=1
# build image trying to use cache image + build cache image:
$ docker build . 
--tag image:latest 
--tag image:build-cache 
--use-cache-from=image:build-cache 
--build-arg BUILDKIT_INLINE_CACHE = 1 
| tee docker.log
# push new image to the registry:
docker push image:latest
# trick: only push cache image to the registry if it was rebuilt:
grep -q CACHED docker.log || docker push image:build-cache

Answer 3

我通过使用Buildah构建映像来解决这个问题。

它可靠且可预测地与缓存一起工作，这与Docker不同(仍然在那些年之后(。

我在 GitLab CI 中使用的代码如下所示：

before_script:
- buildah login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- buildah build
--tag $CI_REGISTRY_IMAGE/app:latest
--cache-from $CI_REGISTRY_IMAGE/app # No tag here
--layers
--cache-to $CI_REGISTRY_IMAGE/app # No tag here
.
- buildah push $CI_REGISTRY_IMAGE/app:latest

缓存存储在独立于应用映像的 GitLab 注册表中。

我花了很多时间研究和测试这个。

还建议将".dockerignore"文件与.git*

使用 Kaniko 是另一个不错的解决方案，但 Buildah 更接近默认的 Docker 命令(1：1 替换(，并且可以轻松添加到默认的 DinD 映像中。