为什么我"google.api_core.exceptions.ResourceExhausted: 429 received trailing metadata size exceeds limit

Question

我是谷歌云平台的新手。我在谷歌顶点AI上传模型后创建了一个端点。但当我运行示例请求中建议的预测函数(python(时，我会收到以下错误：-

Traceback (most recent call last):
File "C:UsersMyanaconda3libsite-packagesgoogleapi_coregrpc_helpers.py", line 67, in 
error_remapped_callable
return callable_(*args, **kwargs)
File "C:UsersMyanaconda3libsite-packagesgrpc_channel.py", line 923, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "C:UsersMyanaconda3libsite-packagesgrpc_channel.py", line 826, in 
_end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "received trailing metadata size exceeds limit"
debug_error_string = "{"created":"@1622724354.768000000","description":"Error received 
from peer ipv4:***.***.***.**","file":"src/core/lib/surface/call.cc", 
"file_line":1063,"grpc_message":"received trailing metadata size exceeds limit", 
"grpc_status":8}">
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "b.py", line 39, in <module>
predict_custom_trained_model_sample(
File "b.py", line 28, in predict_custom_trained_model_sample
response = client.predict(
File "C:UsersMyanaconda3libsite- 
packagesgooglecloudaiplatform_v1servicesprediction_serviceclient.py", line 445, in 
predict
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "C:UsersMyanaconda3libsite-packagesgoogleapi_coregapic_v1method.py", line 145, 
in __call__
return wrapped_func(*args, **kwargs)
File "C:UsersMyanaconda3libsite-packagesgoogleapi_coregrpc_helpers.py", line 69, in 
error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ResourceExhausted: 429 received trailing metadata size exceeds limit

我执行的预测代码是

from typing import Dict
from google.cloud import aiplatform
from google.protobuf import json_format
from google.protobuf.struct_pb2 import Value

def predict_custom_trained_model_sample(
project: str,
endpoint_id: str,
instance_dict: Dict,
location: str = "us-central1",
api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
# The AI Platform services require regional API endpoints.
client_options = {"api_endpoint": api_endpoint}
# Initialize client that will be used to create and send requests.
# This client only needs to be created once, and can be reused for 
multiple requests.
client = 
aiplatform.gapic.PredictionServiceClient(client_options=client_options)
# The format of each instance should conform to the deployed model's prediction input schema.
instance = json_format.ParseDict(instance_dict, Value())
instances = [instance]
parameters_dict = {}
parameters = json_format.ParseDict(parameters_dict, Value())
endpoint = client.endpoint_path(
project=project, location=location, endpoint=endpoint_id
)
response = client.predict(
endpoint=endpoint, instances=instances, parameters=parameters
)
print("response")
print(" deployed_model_id:", response.deployed_model_id)
# The predictions are a google.protobuf.Value representation of the model's predictions.
predictions = response.predictions
for prediction in predictions:
print(" prediction:", dict(prediction)) 

运行此代码后，我得到了错误。如果有人知道这个问题，请帮忙。

Answer 1

需要考虑的几件事：

1. 评测你的自定义容器模型，确保它的预测api函数不是因为某种原因而潜在的
2. 允许您的预测服务使用多个工作者提供服务
3. 增加您在Vertex中的副本数量，或者将您的机器类型设置为更强的类型，只要您有所改进

然而，假设您的大多数预测呼叫都成功通过，并且服务不可用的频率并不高，那么在客户端首先有一些事情值得做，

将您的预测客户端配置为使用Retry(指数退避(：

from google.api_core.retry import Retry, if_exception_type
import requests.exceptions
from google.auth import exceptions as auth_exceptions
from google.api_core import exceptions
if_error_retriable = if_exception_type(
exceptions.GatewayTimeout,
exceptions.TooManyRequests,
exceptions.ResourceExhausted,
exceptions.ServiceUnavailable,
exceptions.DeadlineExceeded,
requests.exceptions.ConnectionError,  # The last three might be an overkill
requests.exceptions.ChunkedEncodingError,
auth_exceptions.TransportError,
)

def _get_retry_arg(settings: PredictionClientSettings):
return Retry(
predicate=if_error_retriable,
initial=1.0, # Initial delay
maximum=4.0, # Maximum delay
multiplier=2.0, # Delay's multiplier
deadline=9.0, # After 9 secs it won't try again and it will throw an exception
)
def predict_custom_trained_model_sample(
project: str,
endpoint_id: str,
instance_dict: Dict,
location: str = "us-central1",
api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
...
response = await client.predict(
endpoint=endpoint,
instances=instances,
parameters=parameters,
timeout=SOME_VALUE_IN_SEC,
retry=_get_retry_arg(),
)