RaggedTensor请求TensorFlow服务失败

Question

我创建了一个使用RaggedTensors的TensorFlow模型。模型工作良好，当调用model.predict时，我得到了预期的结果。

input = tf.ragged.constant([[[-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581], 
[-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963], 
[-0.9984272718429565, -0.942145586013794, 2.48842453956604, -1.6836735010147095, -0.6360141634941101, -1.5785763263702393, -1.900200605392456, -0.49169546365737915, -0.6656315326690674, -0.6583622694015503, -0.5273991227149963], 
]])
model.predict(input)
>> array([[0.5138151 , 0.3277698 , 0.26122513]], dtype=float32)

我已经将模型部署到TensorFlow服务服务器，并使用以下代码进行调用:

import json
import requests
headers = {"content-type": "application/json"}
data = json.dumps({"instances":[
[-1.3523329846758267, ... more data ],
[-1.3523329846758267, ... more data ],
[-1.3523329846758267, ... more data ],
[-1.3523329846758267, ... more data ,
[-1.3523329846758267, ... more data ],
[-1.3523329846758267, ... more data ],
[-1.3523329846758267, ... more data ],
[-1.3523329846758267, ... more data })
json_response = requests.post('http://localhost:8501/v1/models/fashion_model:predict', data=data, headers=headers)
predictions = json.loads(json_response.text)

但是我得到以下错误:

"instances is a plain list, but expecting list of objects as multiple input tensors required as per tensorinfo_map"

我的模型描述:

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['__saved_model_init_op']:
The given SavedModel SignatureDef contains the following input(s):
The given SavedModel SignatureDef contains the following output(s):
outputs['__saved_model_init_op'] tensor_info:
dtype: DT_INVALID
shape: unknown_rank
name: NoOp
Method name is: 
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['args_0'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 11)
name: serving_default_args_0:0
inputs['args_0_1'] tensor_info:
dtype: DT_INT64
shape: (-1)
name: serving_default_args_0_1:0
The given SavedModel SignatureDef contains the following output(s):
outputs['dense_2'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 3)
name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
WARNING: Logging before flag parsing goes to stderr.
W0124 09:33:16.365564 140189730998144 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling __init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
Defined Functions:
Function Name: '__call__'
Option #1
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
Argument #2
DType: bool
Value: True
Argument #3
DType: NoneType
Value: None
Option #2
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
Argument #2
DType: bool
Value: False
Argument #3
DType: NoneType
Value: None
Function Name: '_default_save_signature'
Option #1
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
Function Name: 'call_and_return_all_conditional_losses'
Option #1
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
Argument #2
DType: bool
Value: True
Argument #3
DType: NoneType
Value: None
Option #2
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
Argument #2
DType: bool
Value: False
Argument #3
DType: NoneType
Value: None

我错过了什么?

更新:在检查saved_model_cli输出后，我怀疑我应该像下面这样将请求作为对象发送，但我不确定输入…

{
"instances": [
{
"args_0": nested-list ?,
"args_0_1": ???
}
]
}

更新2用于测试此场景的Colab, Colab中包含下载模型的链接。

更新3:

根据@Niteya Shah的建议，我用:

调用API

data = json.dumps({
"inputs": {
"args_0": [[-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581], 
[-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963]],
"args_0_1": [1, 2]  #Please Check what inputs come here ?
}
})

并得到了结果(终于!):

{'outputs': [[0.466771603, 0.455221593, 0.581544757]]}

然后用相同的数据调用模型，如下所示:

import numpy as np
input = tf.ragged.constant([[
[-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581], 
[-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963]
]])
model.predict(input)

得到了不同的结果:

array([[0.4817084 , 0.3649785 , 0.01603118]], dtype=float32)

我想我还没到。

Answer 1

https://www.tensorflow.org/tfx/serving/api_rest#predict_api

我认为您需要使用REST API中推荐的列格式而不是行格式，因为您的第0个输入的尺寸不匹配。这意味着您将不得不使用输入而不是实例。由于您也有多个输入，因此您还必须将其作为命名输入。

一个示例数据请求可能看起来像这样

data = json.dumps({
"inputs": {
"args_0": [[-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581], 
[-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963]],
"args_0_1": [10, 11]  #Substitute this with the correct row partition values. 
}
})

编辑:

我从这里读到了粗糙张量，似乎第二个输入可能是行分区。我无法在文档中找到它关于什么行分区风格通常使用，所以我使用行长度方法。幸运的是，TensorFlow褴褛为我们提供了这样做的方法。使用values和row_splits属性访问它们。应该可以。

Answer 2

其他人可能会从中受益，因为我花了一些时间拼凑起来:

1. 在粗糙张量上训练玩具LSTM模型。
2. 加载到TensorFlow服务中
3. 使用序列化粗糙张量发出预测请求。

如果有人知道如何重命名"args_0"one_answers"args_0_1"，请添加。相关Git Issue: https://github.com/tensorflow/tensorflow/issues/37226

<标题>构建,保存模型h1> ensorFlow版本:2.9.1Python版本:3.8.12

# Task: predict whether each sentence is a question or not.
sentences = tf.constant(
['What makes you think she is a witch?',
'She turned me into a newt.',
'A newt?',
'Well, I got better.'])
is_question = tf.constant([True, False, True, False])
# Preprocess the input strings.
hash_buckets = 1000
words = tf.strings.split(sentences, ' ')
hashed_words = tf.strings.to_hash_bucket_fast(words, hash_buckets)

# Build the Keras model.
keras_model = tf.keras.Sequential([
tf.keras.layers.Input(shape=[None], dtype=tf.int64, ragged=True),
tf.keras.layers.Embedding(hash_buckets, 16),
tf.keras.layers.LSTM(32, use_bias=False),
tf.keras.layers.Dense(32),
tf.keras.layers.Activation(tf.nn.relu),
tf.keras.layers.Dense(1)
])
keras_model.compile(loss='binary_crossentropy', optimizer='rmsprop')
keras_model.fit(hashed_words, is_question, epochs=5)
print(keras_model.predict(hashed_words))
keras_module_path = "/home/ec2-user/SageMaker/keras-toy-lstm/1"
tf.keras.Model.save(keras_model, keras_module_path)

<标题>

加载,从Model推断将模型加载到TensorFlow服务容器

docker run -t --rm -p 8501:8501 -v "/home/ec2-user/SageMaker/keras-toy-lstm/:/models/keras-model" -e MODEL_NAME=keras-model  tensorflow/serving 

import requests
import json 
payload = {"args_0": [940, 203, 668, 638], 
"args_0_1": [0, 4]}
headers = {"content-type": "application/json"}
data = json.dumps({"inputs":payload})
r = requests.post('http://localhost:8501/v1/models/keras-model:predict', data=data, headers=headers)
r.text

<标题>SavedModelCLI输出

(tensorflow2_p38) sh-4.2$ saved_model_cli show --dir /tmp/tmpgp0loz1v/ --all
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['__saved_model_init_op']:
The given SavedModel SignatureDef contains the following input(s):
The given SavedModel SignatureDef contains the following output(s):
outputs['__saved_model_init_op'] tensor_info:
dtype: DT_INVALID
shape: unknown_rank
name: NoOp
Method name is: 
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['args_0'] tensor_info:
dtype: DT_INT64
shape: (-1)
name: serving_default_args_0:0
inputs['args_0_1'] tensor_info:
dtype: DT_INT64
shape: (-1)
name: serving_default_args_0_1:0
The given SavedModel SignatureDef contains the following output(s):
outputs['dense_1'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 1)
name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
Concrete Functions:
Function Name: '__call__'
Option #1
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)
Argument #2
DType: bool
Value: True
Argument #3
DType: NoneType
Value: None
Option #2
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)
Argument #2
DType: bool
Value: False
Argument #3
DType: NoneType
Value: None
Function Name: '_default_save_signature'
Option #1
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)
Function Name: 'call_and_return_all_conditional_losses'
Option #1
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)
Argument #2
DType: bool
Value: True
Argument #3
DType: NoneType
Value: None
Option #2
Callable with:
Argument #1
DType: RaggedTensorSpec
Value: RaggedTensorSpec(TensorShape([None, None]), tf.int64, 1, tf.int64)
Argument #2
DType: bool
Value: False
Argument #3
DType: NoneType
Value: None