无法在docker构建中从工件注册表安装私有依赖项

我正在尝试安装一个私有python包，该包被上传到docker容器内的工件注册表(在cloudrun上部署它)。

我过去曾在一个云功能中成功地使用过这个包，所以我确信这个包可以工作。

cloudbuild.yaml

steps:
- name: 'gcr.io/cloud-builders/docker'
args: [ 'build', '-t', 'gcr.io/${_PROJECT}/${_SERVICE_NAME}:$SHORT_SHA', '--network=cloudbuild', '.', '--progress=plain']

Dockerfile

FROM python:3.8.6-slim-buster
ENV APP_PATH=/usr/src/app
ENV PORT=8080
# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython 
RUN pip install --upgrade pip
# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH
COPY / $APP_PATH
WORKDIR $APP_PATH
RUN pip install -r requirements.txt --no-color
RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3 # This line is where the bug occurs

# Expose port 
EXPOSE $PORT
# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app

我添加的权限是:

cloudbuild默认服务账号(project-number@cloudbuild.gserviceaccount.com):Artifact Registry Reader
运行cloudbuild的服务帐户:Artifact Registry Reader
运行应用程序的服务账号:Artifact Registry Reader

cloudbuild error:

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 340, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__

从您的回溯日志中，我们可以看到Cloud Build没有凭据来验证私有repo:

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception: //<-ASKING FOR USERNAME

我将一个简单的包上传到一个私有的Artifact Registry仓库，以便在构建容器时进行测试，并且也收到了相同的消息。由于您似乎正在使用服务帐户密钥进行身份验证，因此需要将用户名和密码存储在pip.conf:

中。pip.conf

[global]
extra-index-url = https://_json_key_base64:KEY@LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/

因此，该文件需要在构建过程中可用。多阶段docker构建在这里非常有用，以确保配置密钥不被暴露，因为我们可以选择哪些文件将其放入最终映像中(配置密钥只会在从私有repo下载包时出现): 样本Dockerfile

# Installing packages in a separate image
FROM python:3.8.6-slim-buster as pkg-build
# Target Python environment variable to bind to pip.conf
ENV PIP_CONFIG_FILE /pip.conf
WORKDIR /packages/
COPY requirements.txt /
# Copying the pip.conf key file only during package downloading
COPY ./config/pip.conf /pip.conf
# Packages are downloaded to the /packages/ directory
RUN pip download -r /requirements.txt
RUN pip download --extra-index-url https://LOCATION-python.pkg.dev/PROJECT/REPO/simple/ PACKAGES
# Final image that will be deployed
FROM python:3.8.6-slim-buster
ENV PYTHONUNBUFFERED True
ENV APP_HOME /app
WORKDIR /packages/
# Copying ONLY the packages from the previous build
COPY --from=pkg-build /packages/ /packages/
# Installing the packages from the copied files
RUN pip install --no-index --find-links=/packages/ /packages/*
WORKDIR $APP_HOME
COPY ./src/main.py ./
# Executing sample flask web app 
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

我基于这个相关线程上面的dockerfile，我可以确认这些包是从我的私人Artifact Registry repo正确下载的，并且pip.conf文件也不存在于结果映像中。

最好的方法是在构建时挂载Docker secret。

要做到这一点，您需要将此添加到您的Dockerfile:

# Argument `GOOGLE_APPLICATION_CREDENTIALS` will be set from 
# Docker `/run/secrets/gsa_key` value
ARG GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/gsa_key
# Install the keyring, needed to access Google Artifact Registry.
RUN pip install keyring keyrings.google-artifactregistry.auth
# Mount the secret (!) and install the private package.
RUN --mount=type=secret,id=gsa_key pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3

。，最后的Dockerifle看起来应该或多或少像这样:

FROM python:3.8.6-slim-buster
ENV APP_PATH=/usr/src/app
ENV PORT=8080
# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython 
RUN pip install --upgrade pip
# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH
COPY / $APP_PATH
WORKDIR $APP_PATH
# Argument `GOOGLE_APPLICATION_CREDENTIALS` will be set from 
# Docker `/run/secrets/gsa_key` value
ARG GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/gsa_key
RUN pip install -r requirements.txt --no-color
# Install the keyring, needed to access Google Artifact Registry.
RUN pip install keyring keyrings.google-artifactregistry.auth
# Mount the secret (!) and install the private package.
RUN --mount=type=secret,id=gsa_key pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
# Expose port 
EXPOSE $PORT
# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app

设置好之后，你就可以用下面的命令来构建Docker镜像了:

DOCKER_BUILDKIT=1 docker build 
--secret id=gsa_key,src=/path/to/your/google/credentials.json 
-t my-application .

只要把它调整到你的cloudbuild.yaml，你应该没问题。另外，请注意，您不需要执行多级操作，因为Docker映像中从未显示秘密。

p。d。:你需要一个最新的Docker版本，并且启用Docker Buildkit。

相关内容

最新更新

热门标签：