正在读取App.conf和Log4j. conf.属性从Spark提交命令

Question

我试图通过application.conf和Log4j。这些属性在我的罐子里，但我想把它们传递给外部。我尝试通过spark提交命令，如下所示:

/opt/apps/spark/bin/spark-submit 
--files /opt/apps/spark/code/src/main/resources/application.conf 
--conf spark.driver.extraJavaOptions=-Dconfig.file=/opt/apps/spark/code/src/main/resources/application.conf 
--conf spark.executor.extraJavaOptions=-Dconfig.file=/opt/apps/spark/code/src/main/resources/application.conf 
--driver-java-options Dlog4j.configuration=File:///opt/apps/spark/code/src/main/resources/Log4j.properties 
--class com.ericsson.dnb.main.staging.SnifferReportStatus /opt/apps/spark/code/target/scala-2.12/dnb-reporting-1A-4.0.0.ft.jar

它不会抛出任何错误，但它只读取log4j。属性，并从JAR本身获取application.conf。有没有办法在JAR之外通过Spark提交命令提供这些配置?

Answer 1

我认为这是因为您忘记将file://附加到应用程序配置文件中，而您对log4j做了相同的操作。属性文件。

下面的代码应该可以工作

/opt/apps/spark/bin/spark-submit --files File:///opt/apps/spark/code/src/main/resources/application.conf --conf spark.driver.extraJavaOptions=-Dconfig.file=File:///opt/apps/spark/code/src/main/resources/application.conf --conf spark.executor.extraJavaOptions=-Dconfig.file=File:///opt/apps/spark/code/src/main/resources/application.conf --driver-java-options -Dlog4j.configuration=File:///opt/apps/spark/code/src/main/resources/Log4j.properties --class com.ericsson.dnb.main.staging.SnifferReportStatus /opt/apps/spark/code/target/scala-2.12/dnb-reporting-1A-4.0.0.ft.jar