我希望在我的多阶段Dockerfile中缓存一个特定的阶段，我的测试阶段在努力加快构建过程中使用。否则，它会先进行单元测试，然后再进行集成测试。
下面是Dockerfile的一个基本示例:
# creating a node base
FROM node:16-slim as node-base
ENV CI=true
# builder-base is used to build dependencies
FROM node-base as builder-base
COPY ./package-lock.json ./package.json ./
RUN npm ci --production
# 'development' stage installs all dev deps and can be used to develop code.
FROM builder-base as development
WORKDIR /app
COPY . .
RUN npm ci
EXPOSE 4001
CMD ["npm", "start"]
# 'unit-tests' stage
FROM development AS unit-tests
RUN npm test -- --coverage --testNamePattern=UT:
# 'integration-tests' stage
FROM development AS integration-tests
RUN npm test -- --coverage --testNamePattern=IT:
我想缓存development阶段，pull它，只是运行unit-tests和integration-tests阶段没有构建development两次。
我发现了这个问题，我正试图实现:
如何在Azure DevOps中启用Docker层缓存
上面的答案是:
- task: Docker@2
inputs:
containerRegistry: '$(ContainerRegistryName)'
command: 'login'
- script: "docker pull $(ACR_ADDRESS)/$(REPOSITORY):latest"
displayName: Pull latest for layer caching
continueOnError: true # for first build, no cache
- task: Docker@2
displayName: build
inputs:
containerRegistry: '$(ContainerRegistryName)'
repository: '$(REPOSITORY)'
command: 'build'
Dockerfile: './dockerfile '
buildContext: '$(BUILDCONTEXT)'
arguments: '--cache-from=$(ACR_ADDRESS)/$(REPOSITORY):latest'
tags: |
$(Build.BuildNumber)
latest
- task: Docker@2
displayName: "push"
inputs:
command: push
containerRegistry: "$(ContainerRegistryName)"
repository: $(REPOSITORY)
tags: |
$(Build.BuildNumber)
latest
我已经将它重新用于我的管道，像这样:
# pr.yaml
# # This is triggered by the PR and branch policies
trigger: none
# Specify this to run on the app repo
resources:
repositories:
- repository:app
type: git
name: app
# Read in the base variable template
variables:
- template: templates/variables.yaml
# Use the ubuntu-latest image
pool:
vmIMage: $(vmImageName)
# Stages and their templates for the PR pipeline
stages:
# Checks to see what services in the mono repo have changed by comparing
# the PR code to trunk
- template: templates/changed.yaml
parameters:
comparedTo: origin/trunk
# Run unit tests for each changed service
- template: templates/services.yaml
parameters:
stageName: BuildDev
stageDisplayName: Build dev stage for services...
dockerCommand: build
phrase: build dev
target: development
tag: latest
# Run unit tests for each changed service
- template: templates/services.yaml
parameters:
stageName: UnitTests
stageDisplayName: Run unit tests for services...
dockerCommand: build
phrase: unit test
target: unit-tests
tag: ut-$(Build.BuildNumber)
# Run integration tests for each changed service
- template: templates/services.yaml
parameters:
stageName: IntegrationTests
stageDisplayName: Run integration tests for services...
dockerCommand: build
phrase: integration test
target: integration-tests
tag: it-$(Build.BuildNumber)
# services.yaml
parameters:
- name: stageName
default: ''
- name: stageDisplayName
default: ''
- name: phrase
default: ''
- name: dockerCommand
default: ''
- name: target
default: ''
- name: tag
default: ''
- name: services
type: object
default:
- admin-v2
- api
- portal
stages:
- stage: ${{ parameters.stageName }}
displayName: ${{ parameters.stageDisplayName }}
# Run if detectChanges ran successfully
dependsOn:
- Changed
- ${{ if eq(parameters.stageName, 'UnitTests') }}:
- BuildDev
- ${{ if eq(parameters.stageName, 'IntegrationTests') }}:
- BuildDev
- UnitTests
condition: succeeded()
jobs:
# Runs for all other stages
- ${{ each service in parameters.services }}:
- template: docker.yaml
parameters:
service: ${{ service }}
stageName: ${{ parameters.stageName }}
jobName: ${{ service }}${{ parameters.stageName }}
jobDisplayName: Run ${{ parameters.phrase }} for ${{ service }} service...
taskDisplayName: Run ${{ service }} ${{ parameters.phrase }} tasks...
dockerCommand: ${{ parameters.dockerCommand }}
target: ${{ parameters.target }}
tag: ${{ parameters.tag }}
# docker.yaml
parameters:
- name: stageName
default: ''
- name: service
default: ''
- name: jobName
default: ''
- name: jobDisplayName
default: ''
- name: taskDisplayName
default: ''
- name: dockerCommand
default: ''
- name: target
default: ''
- name: tag
default: ''
jobs:
- job:
displayName: ${{ parameters.jobDisplayName }}
# Handle whether to run for service or not
variables:
servicesChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged'] ]
condition: or(contains(variables['servicesChanged'], '${{ parameters.service }}'), eq(variables['Build.Reason'], 'Manual'))
steps:
# Set to app repo
- checkout: app
# Create mysecrets.txt primarily for Django system check
- bash: |
printenv >> $(dockerFilePath)/${{ parameters.service }}/mysecrets.txt
displayName: Create mysecrets.txt for ${{ parameters.service }}
env:
DJANGO_SECRET_KEY: $(DJANGO_SECRET_KEY)
- ${{ if not(eq(parameters.stageName, 'BuildDev')) }}:
# Run the Docker task
- task: Docker@2
inputs:
containerRegistry: $(dockerRegistryServiceConnection)
command: login
- script: docker pull $(containerRegistry)/$(imageRepository)-${{ parameters.service }}:latest
displayName: Pull latest for layer caching
continueOnError: true # for first build, no cache
- task: Docker@2
# Run if there have been changes
displayName: ${{ parameters.taskDisplayName }}
inputs:
command: ${{ parameters.dockerCommand }}
repository: $(imageRepository)-${{ parameters.service }}
dockerfile: $(dockerFilePath)/${{ parameters.service }}/docker/Dockerfile
buildContext: $(dockerFilePath)/${{ parameters.service }}
containerRegistry: $(dockerRegistryServiceConnection)
arguments: |
--target ${{ parameters.target }}
--cache-from=$(containerRegistry)/$(imageRepository)-${{ parameters.service }}:latest
tags: |
${{ parameters.tag }}-$(Build.BuildNumber)
env:
DOCKER_BUILDKIT: 1
- ${{ if eq(parameters.stageName, 'BuildDev') }}:
- task: Docker@2
# Run if there have been changes
displayName: ${{ parameters.taskDisplayName }}
inputs:
command: ${{ parameters.dockerCommand }}
repository: $(imageRepository)-${{ parameters.service }}
dockerfile: $(dockerFilePath)/${{ parameters.service }}/docker/Dockerfile
buildContext: $(dockerFilePath)/${{ parameters.service }}
containerRegistry: $(dockerRegistryServiceConnection)
arguments: --target ${{ parameters.target }}
tags: |
${{ parameters.tag }}
env:
DOCKER_BUILDKIT: 1
- task: Docker@2
displayName: Pushing ${{ parameters.service }} ${{ parameters.tag }} to ACR
inputs:
command: push
repository: $(imageRepository)-${{ parameters.service }}
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
${{ parameters.tag }}
stageNameBuildDev是development阶段将被建造的地方。
一切运行成功。提取映像的任务显示正在提取映像，然后运行--target unit-tests的任务显示importing cache manifest from ***/app-admin:dev-20211030.11，但它仍然构建development和之前的阶段。它把它拉出来，看到它在那里，然后决定不管怎样都要建造它。
日志如下:
# Pull Job
2021-11-02T23:29:33.0494768Z ##[section]Starting: Pull latest for layer caching
2021-11-02T23:29:33.0502467Z ==============================================================================
2021-11-02T23:29:33.0502844Z Task : Command line
2021-11-02T23:29:33.0503204Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2021-11-02T23:29:33.0503544Z Version : 2.182.0
2021-11-02T23:29:33.0504030Z Author : Microsoft Corporation
2021-11-02T23:29:33.0504416Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2021-11-02T23:29:33.0504832Z ==============================================================================
2021-11-02T23:29:33.2083789Z Generating script.
2021-11-02T23:29:33.2098563Z Script contents:
2021-11-02T23:29:33.2099499Z docker pull ***/app-admin-v2:latest
2021-11-02T23:29:33.2100315Z ========================== Starting Command Output ===========================
2021-11-02T23:29:33.2155259Z [command]/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/5e7263fa-2853-4e6d-b303-62fe80cfacdc.sh
2021-11-02T23:29:34.5925241Z latest: Pulling from app-admin-v2
2021-11-02T23:29:34.5933140Z b380bbd43752: Pulling fs layer
2021-11-02T23:29:34.5933539Z 8d36a6ce056a: Pulling fs layer
2021-11-02T23:29:34.5933881Z f54546b42be1: Pulling fs layer
2021-11-02T23:29:34.5934203Z f5bd69d20a35: Pulling fs layer
2021-11-02T23:29:34.5934568Z 21494383f180: Pulling fs layer
2021-11-02T23:29:34.5934902Z 87500a3a7192: Pulling fs layer
2021-11-02T23:29:34.5935238Z debc4a9f3725: Pulling fs layer
2021-11-02T23:29:34.5935558Z 1b67e176d924: Pulling fs layer
2021-11-02T23:29:34.5935890Z d603a960b591: Pulling fs layer
2021-11-02T23:29:34.5936223Z 9e85221572ee: Pulling fs layer
2021-11-02T23:29:34.5943277Z f5bd69d20a35: Waiting
2021-11-02T23:29:34.5943992Z 21494383f180: Waiting
2021-11-02T23:29:34.5944321Z 87500a3a7192: Waiting
2021-11-02T23:29:34.5944642Z debc4a9f3725: Waiting
2021-11-02T23:29:34.5944958Z 1b67e176d924: Waiting
2021-11-02T23:29:34.5945246Z d603a960b591: Waiting
2021-11-02T23:29:34.5945573Z 9e85221572ee: Waiting
2021-11-02T23:29:34.9794967Z 8d36a6ce056a: Verifying Checksum
2021-11-02T23:29:34.9799329Z 8d36a6ce056a: Download complete
2021-11-02T23:29:35.6335949Z f5bd69d20a35: Verifying Checksum
2021-11-02T23:29:35.6337075Z f5bd69d20a35: Download complete
2021-11-02T23:29:35.8084539Z b380bbd43752: Verifying Checksum
2021-11-02T23:29:35.8120185Z b380bbd43752: Download complete
2021-11-02T23:29:35.9459756Z 21494383f180: Verifying Checksum
2021-11-02T23:29:35.9460303Z 21494383f180: Download complete
2021-11-02T23:29:35.9926957Z f54546b42be1: Verifying Checksum
2021-11-02T23:29:35.9927391Z f54546b42be1: Download complete
2021-11-02T23:29:36.1820048Z 87500a3a7192: Verifying Checksum
2021-11-02T23:29:36.1820481Z 87500a3a7192: Download complete
2021-11-02T23:29:36.4513965Z 1b67e176d924: Verifying Checksum
2021-11-02T23:29:36.4514451Z 1b67e176d924: Download complete
2021-11-02T23:29:37.1143768Z d603a960b591: Verifying Checksum
2021-11-02T23:29:37.1144264Z d603a960b591: Download complete
2021-11-02T23:29:37.3920871Z b380bbd43752: Pull complete
2021-11-02T23:29:38.0559222Z 9e85221572ee: Verifying Checksum
2021-11-02T23:29:38.0559774Z 9e85221572ee: Download complete
2021-11-02T23:29:38.5139277Z debc4a9f3725: Verifying Checksum
2021-11-02T23:29:38.5140203Z debc4a9f3725: Download complete
2021-11-02T23:29:39.0212051Z 8d36a6ce056a: Pull complete
2021-11-02T23:29:40.9828384Z f54546b42be1: Pull complete
2021-11-02T23:29:41.1410341Z f5bd69d20a35: Pull complete
2021-11-02T23:29:41.2067833Z 21494383f180: Pull complete
2021-11-02T23:29:41.2833611Z 87500a3a7192: Pull complete
2021-11-02T23:29:54.5480084Z debc4a9f3725: Pull complete
2021-11-02T23:29:54.6097840Z 1b67e176d924: Pull complete
2021-11-02T23:29:54.6823771Z d603a960b591: Pull complete
2021-11-02T23:30:04.3756447Z 9e85221572ee: Pull complete
2021-11-02T23:30:04.3801963Z Digest: sha256:64308db1d461a2aff0deaf31b5bb5694becfb2298f0c474366d1d9b695b0a441
2021-11-02T23:30:04.3836112Z Status: Downloaded newer image for ***/app-admin-v2:latest
2021-11-02T23:30:04.3874744Z ***/app-admin-v2:latest
2021-11-02T23:30:04.4026099Z ##[section]Finishing: Pull latest for layer caching
# Build Job
2021-11-02T23:30:08.6626317Z [command]/usr/bin/docker build -f /home/vsts/work/1/s/admin-v2/docker/Dockerfile --label com.azure.dev.image.system.teamfoundationcollectionuri=https://dev.azure.com/thecompany/ --label com.azure.dev.image.system.teamproject=-dev --label com.azure.dev.image.build.repository.name=production-resources --label com.azure.dev.image.build.sourceversion=809969a7bb0880a135c935c5d66ea0e2bba2c65e --label com.azure.dev.image.build.repository.uri=https://thecompany@dev.azure.com/thecompany/-dev/_git/production-resources --label com.azure.dev.image.build.sourcebranchname=main --label com.azure.dev.image.build.definitionname= App PR --label com.azure.dev.image.build.buildnumber=20211102.3 --label com.azure.dev.image.build.builduri=vstfs:///Build/Build/1322 --label image.base.ref.name=nginx --label image.base.digest=sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36 --target unit-tests --cache-from=***/app-admin-v2:latest -t ***/app-admin-v2:ut-20211102.3-20211102.3 /home/vsts/work/1/s/admin-v2
2021-11-02T23:30:08.9603622Z #1 [internal] load build definition from Dockerfile
2021-11-02T23:30:08.9604133Z #1 sha256:7818e7e1291667e1af9ce6f8a463d74e3df7b64001596dd00b927ccc12c37515
2021-11-02T23:30:08.9604573Z #1 transferring dockerfile: 1.05kB done
2021-11-02T23:30:08.9604904Z #1 DONE 0.0s
2021-11-02T23:30:08.9605027Z
2021-11-02T23:30:08.9605303Z #2 [internal] load .dockerignore
2021-11-02T23:30:08.9605713Z #2 sha256:1c18692a923cc3e70a98431aefc897940bf0c36edf7ae0f3b0b525b4d753b7fb
2021-11-02T23:30:08.9606117Z #2 transferring context: 329B done
2021-11-02T23:30:08.9607436Z #2 DONE 0.0s
2021-11-02T23:30:08.9607554Z
2021-11-02T23:30:08.9608532Z #3 [internal] load metadata for docker.io/library/node:16-slim
2021-11-02T23:30:08.9609018Z #3 sha256:faa605aa367b596b57bbdc1bdcccade69c92d97d03d44e595a34c7e28b8d594e
2021-11-02T23:30:10.1243750Z #3 DONE 1.2s
2021-11-02T23:30:10.1243963Z
2021-11-02T23:30:10.1245311Z #12 importing cache manifest from ***/app-admin-v2:latest
2021-11-02T23:30:10.1245833Z #12 sha256:91a404777043fddf396dcad59a4fd8b976e0224c77d860e00947ec3630b83eaf
2021-11-02T23:30:10.1246215Z #12 DONE 0.0s
2021-11-02T23:30:10.1246341Z
2021-11-02T23:30:10.1246618Z #4 [internal] load build context
2021-11-02T23:30:10.1247014Z #4 sha256:e1bc60dd1feb9ca2db3a89f6b76ec616e7b56215986652ad691e1fc4c108a5aa
2021-11-02T23:30:10.1247447Z #4 transferring context: 713.43kB 0.0s done
2021-11-02T23:30:10.1247777Z #4 DONE 0.0s
2021-11-02T23:30:10.1247897Z
2021-11-02T23:30:10.1254657Z #11 [node-base 1/1] FROM docker.io/library/node:16-slim@sha256:9ec1ff69c844f2de3a6a2180cd49ca75797d9f2a0fc52bb33c8a672fd0fe7e18
2021-11-02T23:30:10.1255382Z #11 sha256:af5e5b9d07d96a94506820483ad64d714f1bf9a0e5ae75b7b4e7265902c9f941
2021-11-02T23:30:10.1256263Z #11 resolve docker.io/library/node:16-slim@sha256:9ec1ff69c844f2de3a6a2180cd49ca75797d9f2a0fc52bb33c8a672fd0fe7e18 done
2021-11-02T23:30:10.1256894Z #11 sha256:9ec1ff69c844f2de3a6a2180cd49ca75797d9f2a0fc52bb33c8a672fd0fe7e18 1.21kB / 1.21kB done
2021-11-02T23:30:10.1257454Z #11 sha256:ed230d53c9d9820caa9b1bea418c1f835d15ec4d1253160e908ff31fe074ac35 1.37kB / 1.37kB done
2021-11-02T23:30:10.1258009Z #11 sha256:dd74f260f56dccc771f512ef5b2a81345e3bcefcac34c248459da36169be36b2 6.89kB / 6.89kB done
2021-11-02T23:30:10.1258404Z #11 DONE 0.1s
2021-11-02T23:30:10.2748114Z
2021-11-02T23:30:10.2749533Z #5 [builder-base 1/2] COPY ./package-lock.json ./package.json ./
2021-11-02T23:30:10.2774757Z #5 sha256:e5803272aeaf38a29bf5ca34e14cc0c613c673044fdab9a2c27d837bb2de837c
2021-11-02T23:30:10.2775425Z #5 DONE 0.0s
2021-11-02T23:30:10.2775692Z
2021-11-02T23:30:10.2776552Z #6 [builder-base 2/2] RUN npm ci --production
2021-11-02T23:30:10.2777798Z #6 sha256:30a22a0c38f1eebba7d10ab9f4cd8f24ce6a4599827350e6b42f38836af226ab
2021-11-02T23:30:12.6879927Z #6 2.477 npm WARN old lockfile
2021-11-02T23:30:12.6881199Z #6 2.478 npm WARN old lockfile The package-lock.json file was created with an old version of npm,
2021-11-02T23:30:12.6881732Z #6 2.478 npm WARN old lockfile so supplemental metadata must be fetched from the registry.
2021-11-02T23:30:12.6882137Z #6 2.478 npm WARN old lockfile
2021-11-02T23:30:12.6883737Z #6 2.479 npm WARN old lockfile This is a one-time fix-up, please be patient...
2021-11-02T23:30:12.6884195Z #6 2.479 npm WARN old lockfile
...
知道为什么会发生这种情况以及如何预防吗?真的是在拖慢节奏。
我也有同样的问题。
简单Docker文件:
FROM node:erbium as base
WORKDIR /app
COPY . .
FROM base as test
RUN npm i
RUN npm run test
FROM base as prod
RUN npm ci
EXPOSE 3000
CMD ["node", "app.node.js"]
示例管道:
jobs:
- job: test
displayName: "test before build"
steps:
- task: Docker@2
displayName: 'test service inside docker'
inputs:
command: build
arguments: '--target test'
- job: build
displayName: "build and push to registry"
dependsOn: test
steps:
- task: Docker@2
displayName: 'build and tag'
inputs:
command: build
dockerfile: '**/Dockerfile'
buildContext: '**'
arguments: '--target prod'
repository: 'myName/myImage'
tags: |
dev
$(Build.SourceVersion)
当我有一个不同的作业/任务设置时，我的--target被忽略。
解决方案:
最后，我把我的Dockerfile分成了2个文件。
Dockerfile-test
FROM node:erbium as base
WORKDIR /app
COPY . .
FROM base as test
RUN npm i
RUN npm run test
Dockerfile
FROM node:erbium as base
WORKDIR /app
COPY . .
FROM base as prod
RUN npm ci
EXPOSE 3000
CMD ["node", "app.node.js"]
Azure Pipeline Docker Caching根本无法工作-节省您的时间尝试修复它
我建议在docker中构建你的映像——这将允许你在任何地方运行，而不依赖于特定的云供应商和他们的错误
我在这里问了一个相关的问题后，设法使这个工作:
解决Azure Pipeline中Docker层缓存不工作的问题