我登录了(gcloud auth login)，但除此之外，我没有详细说明为什么这个命令不起作用。代码:
from google.cloud import aiplatform
job = aiplatform.PipelineJob(display_name="MY_DISPLAY_JOB", template_path=/tmp/tmpt727de81, project=project_id, pipeline_root=project_gcp_bucket)
job.submit()
错误:
Creating PipelineJob
Traceback (most recent call last):
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})"
debug_error_string = "{"created":"@1650655555.692528693","description":"Error received from peer us-central1-aiplatform.googleapis.com:443","file":"src/core/lib/surface/call.cc","file_line":905,"grpc_message":"Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})","grpc_status":14}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/daaronch/code/same-project/sameproject/ops/vertex/vertex_debugger.py", line 108, in <module>
main(auto_envvar_prefix="SAME")
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/daaronch/code/same-project/sameproject/ops/vertex/vertex_debugger.py", line 99, in deploy_vertex
job.submit()
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/google/cloud/aiplatform/pipeline_jobs.py", line 304, in submit
self._gca_resource = self.api_client.create_pipeline_job(
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/google/cloud/aiplatform_v1/services/pipeline_service/client.py", line 1281, in create_pipeline_job
response = rpc(
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
return wrapped_func(*args, **kwargs)
File "/tmp/tmpt727de81/.venv/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 Getting metadata from plugin failed with error: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
我可以重现创建管道。
警告我完全不熟悉顶点AI，所以这是我只是得到任何部署的结果。
我使用Sascha的例子(谢谢Sascha!)来生成一个(ny)pipeline.json:
pipeline.py:
import kfp
from kfp.v2.dsl import pipeline
from kfp.v2.dsl import component
from kfp.v2 import compiler
from os import getenv
from typing import NamedTuple
bucket = getenv("BUCKET")
@component()
def concat(a: str, b: str) -> str:
return a + b
@component
def reverse(a: str)-> NamedTuple("outputs", [("before", str), ("after", str)]):
return a, a[::-1]
@component(output_component_file="component.yaml")
@pipeline(
name="my-pipeline",
description="My ML Pipelines",
pipeline_root=f"gs://{bucket}")
def my_pipeline(a: str="stres", b: str="sed"):
concat_task = concat(a, b)
# Per the example but `reverse_task` variable is redundant
reverse_task = reverse(concat_task.output)
compiler.Compiler().compile(
pipeline_func=my_pipeline,
package_path="my-pipeline.json",
pipeline_name="my-pipeline"
)
运行上述命令生成component.yaml和my-pipeline.json。后者是第二步所需要的，它反映了你的问题中的代码:
main.py:
from google.cloud import aiplatform
from os import getenv
project = getenv("PROJECT")
bucket = getenv("BUCKET")
job = aiplatform.PipelineJob(
display_name="My_Display_Job",
template_path="my-pipeline.json",
project=project,
pipeline_root=f"gs://{bucket}",
parameter_values={
"a":"foo",
"b":"bar",
},
)
job.submit()
Creating PipelineJob
PipelineJob created. Resource name: projects/[REDACTED]/locations/us-central1/pipelineJobs/my-pipeline-20220423110750
To use this PipelineJob in another session:
pipeline_job = aiplatform.PipelineJob.get('projects/[REDACTED]/locations/us-central1/pipelineJobs/my-pipeline-20220423110750')
View Pipeline Job:
https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/my-pipeline-20220423110750?project=[REDACTED]
我不确定是否有一个gcloud命令枚举管道，但我验证了它是在控制台中创建的。
PROJECT=
BUCKET=${PROJECT}
ACCOUNT="tester"
EMAIL="${ACCOUNT}@${PROJECT}.iam.gserviceaccount.com"
gcloud projects create ${PROJECT}
gcloud beta billing projects link ${PROJECT}
--billing-account=${BILLING}
# I think Compute is needed too
for SERVICE in "compute" "aiplatform"
do
gcloud services enable ${SERVICE}.googleapis.com
--project=${PROJECT}
done
gcloud iam service-accounts create ${ACCOUNT}
--project=${PROJECT}
gcloud iam service-accounts keys create ${PWD}/${ACCOUNT}.json
--iam-account=${EMAIL}
# Going big because I'm unsure what role is required
gcloud projects add-iam-policy-binding ${PROJECT}
--member=serviceAccount:${EMAIL}
--role=roles/editor
# Required for the pipeline
gsutil mb -p ${PROJECT} gs://${BUCKET}
# Grant Compute Engine Service Account admin on BUCKET
NUM=$(
gcloud projects describe ${PROJECT}
--format="value(projectNumber)")
gsutil iam ch
serviceAccount:${NUM}-compute@developer.gserviceaccount.com:admin
gs://${BUCKET}
export GOOGLE_APPLICATION_CREDENTIALS=${PWD}/${ACCOUNT}.json
export PROJECT
export BUCKET
python3 -m venv venv
source venv/bin/activate
python3 -m pip install kfp
python3 -m pip install google-cloud-aiplatform
# Generate `my-pipeline.json`
python3 pipeline.py
# Create the pipeline
python3 main.py