涉及 Flink 程序中矩阵操作的错误



flink 程序在本地正常运行,但打包后上传到服务器总是失败,并显示以下消息:

java.lang.NoClassDefFoundError: org/netlib/blas/Dgemm
at com.github.fommil.netlib.F2jBLAS.dgemm(F2jBLAS.java:96)
at breeze.linalg.operators.DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$.apply(DenseMatrixOps.scala:63)
at breeze.linalg.operators.DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$.apply(DenseMatrixOps.scala:48)
at breeze.linalg.ImmutableNumericOps$class.$times(NumericOps.scala:135)
at breeze.linalg.DenseMatrix.$times(DenseMatrix.scala:53)
at com.zte.flink.machinelearning.robustRegressionAlgorithm$.IRLS(robustRegressionAlgorithm.scala:32)
at com.zte.flink.machinelearning.robustRegressionAlgorithm$.predict(robustRegressionAlgorithm.scala:88)
at com.zte.flink.test$.main(test.scala:21)
at com.zte.flink.test.main(test.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
Caused by: java.lang.ClassNotFoundException: org.netlib.blas.Dgemm
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 26 more

org.netlib.blas.Dgemm 似乎位于本地 CLASSPATH 中,但不在服务器的 CLASSPATH 中。你需要构建一个包含这个库的胖 jar,然后将 far jat 提交到集群(推荐(,或者把它放在所有 flink 服务器的 lib 目录中。

有关如何处理 Flink 应用程序的依赖项的更多详细信息,请参阅文档。

相关内容

  • 没有找到相关文章

最新更新