我正在尝试使用Tensorflow Serving project从HDFS中提供Tensorflow模型。
我正在运行张量流服务 docker 容器标签 1.10.1 https://hub.docker.com/r/tensorflow/serving
我可以看到 tensorflow/serve repo 引用 Hadoop 在 https://github.com/tensorflow/serving/blob/628702e1de1fa3d679369e9546e7d74fa91154d3/tensorflow_serving/model_servers/BUILD#L341
"@org_tensorflow//tensorflow/core/platform/hadoop:hadoop_file_system"
这是对
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/hadoop/hadoop_file_system.cc
我设置了以下环境变量:
- HADOOP_HDFS_HOME指向我的HDFS家(在我的情况下为/etc/hadoop(。
- MODEL_BASE_PATH设置为"hdfs://tensorflow/models">
- MODEL_NAME设置为我要加载的模型名称
我将Hadoop home挂载到docker容器中,并可以使用docker exec进行验证。
当我运行 docker 容器时,我在日志中得到以下内容:
tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:369] FileSystemStoragePathSource encountered a file-system access error: Could not find base path hdfs://tensorflow/models/my_model for servable my_model
我发现了Tensorflow使用HDFS进行训练的例子,但没有使用Tensorflow Serving从HDFS中提供模型。
Tensorflow Serving 可以服务于 HDFS 的模型吗? 如果是这样,你怎么做?
在model_servers
BUILD
,在get_model_status_impl_test
的cc_test
下,添加此行@org_tensorflow//tensorflow/core/platform/hadoop:hadoop_file_system
,如下所示:
cc_test(
name = "get_model_status_impl_test",
size = "medium",
srcs = ["get_model_status_impl_test.cc"],
data = [
"//tensorflow_serving/servables/tensorflow/testdata:saved_model_half_plus_two_2_versions",
],
deps = [
":get_model_status_impl",
":model_platform_types",
":platform_config_util",
":server_core",
"//tensorflow_serving/apis:model_proto",
"//tensorflow_serving/core:availability_preserving_policy",
"//tensorflow_serving/core/test_util:test_main",
"//tensorflow_serving/servables/tensorflow:saved_model_bundle_source_adapter_proto",
"//tensorflow_serving/servables/tensorflow:session_bundle_config_proto",
"//tensorflow_serving/servables/tensorflow:session_bundle_source_adapter_proto",
"//tensorflow_serving/test_util",
"@org_tensorflow//tensorflow/cc/saved_model:loader",
"@org_tensorflow//tensorflow/cc/saved_model:signature_constants",
"@org_tensorflow//tensorflow/contrib/session_bundle",
"@org_tensorflow//tensorflow/core:test",
"@org_tensorflow//tensorflow/core/platform/hadoop:hadoop_file_system",
],
)
我认为这将解决您的问题。
参考:无法从 HDFS 加载模型