AWS Sagemaker是否支持gRPC预测请求

Question

我在本地模式下从估计器部署了Sagemaker的Tensorflow模型，当试图使用gRPC调用Tensorflow Serving(TFS(预测端点时，我得到了错误：

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"

我正在做gRPC请求，就像在这篇博客文章中一样：

import grpc from tensorflow.compat.v1 
import make_tensor_protofrom tensorflow_serving.apis 
import predict_pb2from tensorflow_serving.apis 
import prediction_service_pb2_grpc
grpc_port = 9000 # Tried also with other ports such as 8500
request = predict_pb2.PredictRequest()
request.model_spec.name = 'model'
request.model_spec.signature_name = 'serving_default'
request.inputs['input_tensor'].CopyFrom(make_tensor_proto(instance))
options = [
('grpc.enable_http_proxy', 0),
('grpc.max_send_message_length', MAX_GRPC_MESSAGE_LENGTH),
('grpc.max_receive_message_length', MAX_GRPC_MESSAGE_LENGTH)
]
channel = grpc.insecure_channel(f'0.0.0.0:{grpc_port}', options=options)
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
result_future = stub.Predict.future(request, 30)  
output_tensor_proto = result_future.result().outputs['predictions']
output_shape = [dim.size for dim in output_tensor_proto.tensor_shape.dim]
output_np = np.array(output_tensor_proto.float_val).reshape(output_shape)
prediction_json = {'predictions': output_np.tolist()}

查看运行TFS的Sagemaker的docker容器，我在日志中看到rest端点被导出/暴露，但不是gRPC端点，尽管它似乎正在运行：

ensorflow_serving/model_servers/server.cc:417] Running gRPC ModelServer at 0.0.0.0:9000 ...

与gRPC不同，在容器日志中，我可以看到rest端点被导出：

tensorflow_serving/model_servers/server.cc:438] Exporting HTTP/REST API at:localhost:8501 ...

Sagemaker TFS容器甚至支持gRPC吗？如何使用Sagemaker发出gRPC TFS预测请求？

Answer 1

SageMaker端点是REST端点。但是，您可以在容器中建立gRPC连接。不能通过gRPC进行InvokeEndpoint API调用。

如果使用SageMaker TensorFlow容器，则需要传递一个推理.py脚本，该脚本包含向TFS发出gRPC请求的逻辑。

请参阅这个针对TensorFlow Serving进行gRPC预测的示例推理.py脚本。