虽然链接到库,但对函数的引用未定义



我正试图在Docker容器中构建开源项目转换器。这是一个建立在ngraph之上的工具,通过ngraph桥与tensorflow连接,可以使用基于图的抽象编写深度学习程序。transformer是ngraph的扩展,它允许使用同态加密对加密数据进行推理。

我在master分支上使用了最新的he transformer commit(6d5a5b39(。构建过程抛出许多undefined reference错误:

#29 282.5 /usr/bin/clang++-9   -O2 -Wformat -Wformat-security -D_FORTIFY_SOURCE=2 -fstack-protector-all -Wall -march=native -Wno-deprecated-declarations -fopenmp=libomp  -O2 -Wformat -Wformat-security -D_FORTIFY_SOURCE=2 -fstack-protector-all -Wall -march=native -Wno-deprecated-declarations -g   -pie -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_subtract_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_slice_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_softmax_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_server_client_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_reverse_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_result_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_reshape_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_relu_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_read_write_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_power_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_pad_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_negate_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_max_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_multiply_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_minimum_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_maxpool_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_layers_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_rescale_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_dot_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_exp_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_divide_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_convolution_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_constant_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_concat_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_broadcast_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_basics_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_batch_norm_inference_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_avg_pool_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/test_add_HE_SEAL.cpp.o CMakeFiles/unit-test.dir/main.cpp.o CMakeFiles/unit-test.dir/test_seal.cpp.o CMakeFiles/unit-test.dir/test_encryption_parameters.cpp.o CMakeFiles/unit-test.dir/test_he_op_annotations.cpp.o CMakeFiles/unit-test.dir/test_he_plaintext.cpp.o CMakeFiles/unit-test.dir/test_he_tensor.cpp.o CMakeFiles/unit-test.dir/test_he_type.cpp.o CMakeFiles/unit-test.dir/test_he_util.cpp.o CMakeFiles/unit-test.dir/test_he_fusion.cpp.o CMakeFiles/unit-test.dir/test_he_supported_ops.cpp.o CMakeFiles/unit-test.dir/test_propagate_he_annotations.cpp.o CMakeFiles/unit-test.dir/test_he_seal_executable.cpp.o CMakeFiles/unit-test.dir/test_bounded_relu.cpp.o CMakeFiles/unit-test.dir/test_perf_micro.cpp.o CMakeFiles/unit-test.dir/test_protobuf.cpp.o CMakeFiles/unit-test.dir/test_seal_plaintext_wrapper.cpp.o CMakeFiles/unit-test.dir/test_seal_util.cpp.o CMakeFiles/unit-test.dir/test_tcp_message.cpp.o CMakeFiles/unit-test.dir/test_tcp_client.cpp.o CMakeFiles/unit-test.dir/test_ngraph_he_log.cpp.o  -o unit-test -Wl,-rpath,/home/he-transformer/build/src:/home/he-transformer/build/ext_ngraph/src/ext_ngraph-build/src/ngraph:/home/he-transformer/build/ext_zlib/src/ext_zlib-build ../ext_ngraph/src/ext_ngraph-build/test/util/libngraph_test_util.a -lpthread ../src/libhe_seal_backend.so ../protobuf/lib/libprotobuf.a ../gtest/build/googlemock/gtest/libgtest.a ../ext_ngraph/src/ext_ngraph-build/src/ngraph/libngraph.so ../external/lib/libseal-3.4.a ../ext_zlib/src/ext_zlib-build/libz.so 
#29 282.6 INFO: From Compiling tensorflow/lite/kernels/mfcc.cc:
#29 282.6 In file included from ./tensorflow/lite/kernels/internal/common.h:42:0,
#29 282.6                  from ./tensorflow/lite/kernels/cpu_backend_gemm_custom_gemv.h:41,
#29 282.6                  from ./tensorflow/lite/kernels/cpu_backend_gemm.h:22,
#29 282.6                  from ./tensorflow/lite/kernels/internal/optimized/optimized_ops.h:42,
#29 282.6                  from tensorflow/lite/kernels/mfcc.cc:21:
#29 282.6 external/arm_neon_2_x86_sse/NEON_2_SSE.h: In function 'void vst1q_lane_f32(float32_t*, float32x4_t, int)':
#29 282.6 external/arm_neon_2_x86_sse/NEON_2_SSE.h:9725:31: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
#29 282.6      *(ptr) =  *((float*)&ilane);
#29 282.6                                ^
#29 282.6 external/arm_neon_2_x86_sse/NEON_2_SSE.h: In function 'float32_t vgetq_lane_f32(float32x4_t, int)':
#29 282.6 external/arm_neon_2_x86_sse/NEON_2_SSE.h:11964:22: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
#29 282.6      return *(float*)&ilane;
#29 282.6                       ^
#29 283.3 /home/he-transformer/build/test/test_sum_HE_SEAL.cpp:35: undefined reference to `ngraph::runtime::Backend::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)'
#29 283.3 /home/he-transformer/build/test/test_sum_HE_SEAL.cpp:51: undefined reference to `ngraph::Node::get_name[abi:cxx11]() const'
#29 283.3 CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o: In function `bool ngraph::runtime::he::test::all_close<float>(std::vector<float, std::allocator<float> > const&, std::vector<float, std::allocator<float> > const&, float)':
#29 283.3 /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:264: undefined reference to `ngraph::default_logger_handler_func(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 283.3 CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o: In function `bool ngraph::runtime::he::test::all_close<float>(std::vector<float, std::allocator<float> > const&, std::vector<float, std::allocator<float> > const&, float)':
#29 283.3 /home/he-transformer/test/./test_util.hpp:57: undefined reference to `ngraph::LogHelper::LogHelper(ngraph::LOG_TYPE, char const*, int, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)'
#29 283.3 CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o: In function `void __gnu_cxx::new_allocator<ngraph::Function>::construct<ngraph::Function, std::shared_ptr<ngraph::op::v0::Sum>&, std::vector<std::shared_ptr<ngraph::op::v0::Parameter>, std::allocator<std::shared_ptr<ngraph::op::v0::Parameter> > > >(ngraph::Function*, std::shared_ptr<ngraph::op::v0::Sum>&, std::vector<std::shared_ptr<ngraph::op::v0::Parameter>, std::allocator<std::shared_ptr<ngraph::op::v0::Parameter> > >&&)':
#29 283.3 /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/ext/new_allocator.h:136: undefined reference to `ngraph::Function::Function(std::shared_ptr<ngraph::Node> const&, std::vector<std::shared_ptr<ngraph::op::v0::Parameter>, std::allocator<std::shared_ptr<ngraph::op::v0::Parameter> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 283.3 CMakeFiles/unit-test.dir/test_sum_HE_SEAL.cpp.o: In function `__cxx_global_var_init':
#29 283.3 /home/he-transformer/build/test/test_sum_HE_SEAL.cpp:65: undefined reference to `ngraph::combine_test_backend_and_case(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 283.3 /home/he-transformer/build/test/test_sum_HE_SEAL.cpp:65: undefined reference to `ngraph::prepend_disabled(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
...
#29 285.2 ../src/libhe_seal_backend.so: undefined reference to `ngraph::Node::get_provenance_tags[abi:cxx11]() const'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::file_util::read_file_to_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::to_lower(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::runtime::Backend::get_backend_op(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ...)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::runtime::BackendManager::register_backend(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<std::shared_ptr<ngraph::runtime::Backend> (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::file_util::exists(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::descriptor::Tensor::get_name[abi:cxx11]() const'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::descriptor::Tensor::Tensor(ngraph::element::Type const&, ngraph::PartialShape const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::split(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char, bool)'
#29 285.3 ../src/libhe_seal_backend.so: undefined reference to `ngraph::Node::description[abi:cxx11]() const'
...
#29 285.3 clang: error: linker command failed with exit code 1 (use -v to see invocation)
#29 285.3 make[2]: *** [test/unit-test] Error 1
#29 285.3 test/CMakeFiles/unit-test.dir/build.make:825: recipe for target 'test/unit-test' failed
#29 285.3 make[2]: Leaving directory '/home/he-transformer/build'
#29 285.3 make[1]: *** [test/CMakeFiles/unit-test.dir/all] Error 2

请参阅完整日志的原始输出,以及我用于构建转换器的Dockerfile的存储库。(注意:我知道transformer本身也提供了Dockerfile+Makefile,但它们不适合我。然而,我的是基于他们的Ubuntu 18.04 Dockerfile。(

我对cmake不太熟悉,该项目的构建设置非常复杂,有很多依赖项,但听起来libhe_seal_backend.so没有正确链接到同样作为外部项目构建的ngraph,请参阅ngraph.cmake。然而,正如你在CMakeLists.txt中看到的那样,它是链接的,因为libhe_seal_backend目标是构建的。

由于目标unit-test导致问题,我认为可能需要将其与ngraph链接。为此,我添加了相应的链接目标,并用修改后的CMakeLists.txt替换了克隆转换器存储库后的文件。不幸的是,这也引发了同样的错误。

注意:如果您试图复制它,我花了大约45分钟在48核AWS机器上构建它,因为它还将tensorflow作为外部项目构建。

感谢transformer的存储库维护人员Fabian Boemer的这一提示。他注意到日志显示CXX ABI存在问题。该问题是由使用旧版本的g++(<5.1(编译ngraph引起的。日志证实了这一点,请参阅compiler identification is GNU 4.8.5

从Dockerfile中删除gcc 4.8的安装,使系统默认为gcc-7,解决了这个问题,并可以成功编译transformer。

最新更新