如何在谷歌cloudbuild步骤中持久化变量

Question

我有一个cloudbuild.json，用于将管道上传到gcp kubeflow。现在，我想添加另一个步骤，在该步骤中，我想获取最新的管道id，然后将管道作为实验运行。因此，我的主要问题是如何在后续步骤中获得管道id。我已经写了一个小脚本来获取最新的管道id，并将其添加为从docker运行的步骤，但现在我不确定如何获取该管道id。

这是我的coudbuild.json

{
"steps": [
{
"name": "gcr.io/cloud-builders/docker",
"args": [
"build",
"-t",
"trainer_image",
"."
],
"dir": "./trainer_image/"
},
{
"name": "gcr.io/cloud-builders/docker",
"args": [
"build",
"-t",
"base_image",
"."
],
"dir": "./base_image/"
},
{
"name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
"args": [
"-c",
"dsl-compile --py covertype_training_pipeline.py --output covertype_training_pipeline.yaml"
],
"env": [
"BASE_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test",
"TRAINER_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
"RUNTIME_VERSION=1.15",
"PYTHON_VERSION=3.7",
"COMPONENT_URL_SEARCH_PREFIX=https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/",
"USE_KFP_SA=False"
],
"dir": "./pipeline/"
},
{
"name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
"args": [
"-c",
"kfp --endpoint 66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com pipeline upload -p credit_fraud_training_test covertype_training_pipeline.yaml"
],
"dir": "./pipeline/"
},
{
"name": "gcr.io/cloud-builders/docker",
"args": [
"build",
"-t",
"id_image",
"."
],
"dir": "./id_image/"
}
],
"images": [
"gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
"gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test"
]
}

这是我的python脚本，用于获取最新的管道id

import kfp
client = kfp.Client(host='66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com')
pipelines = client.list_pipelines()
total_pipeline = len(pipelines.pipelines)
latest_pipeline_id = pipelines.pipelines[total_pipeline-1].id
with open('/workspace/pipeline.txt') as file:
file.write(latest_pipeline_id)

Answer 1

不能在步骤之间持久化变量。您只能持久化文件。但是，在步骤中，您可以重用局部变量。

- name: "gcr.io/cloud-builders/docker",
entrypoint: "bash"
args: 
- "-c"
- |
export DOCKER_VERSION=$(docker version)
echo $$DOCKER_VERSION #Reuse local variable
echo $$DOCKER_VERSION > DOCKER_VERSION.txt #Save the variable for the next step
- name: "gcr.io/cloud-builders/docker",
entrypoint: "bash"
args: 
- "-c"
- |
export DOCKER_VERSION=$(cat DOCKER_VERSION.txt) #Get the value saved in file
echo $$DOCKER_VERSION #Reuse local variable