火花提交信息日志

Question

我正在使用火花 1.2.1

：

/home/ubuntu/spark-1.2.1/bin/spark-submit --master spark://11.11.11.11:7077 --driver-class-path /home/ubuntu/spark-cassandra-connector-java-assembly-1.2.1-FAT.jar --class "$class" "$jar

我在输出中得到很多日志：

> 16/03/14 09:59:53 INFO AppClient$ClientActor: Connecting to master
> spark://11.11.11.11:7077... 16/03/14 09:59:54 INFO
> SparkDeploySchedulerBackend: Connected to Spark cluster with app ID
> app-20160314095954-0001 16/03/14 09:59:54 INFO AppClient$ClientActor:
> Executor added: app-20160314095954-0001/0 on
> worker-20160314095247-ip-11.11.11.11.ec2.internal-54560 (ip
> -11.11.11.11.ec2.internal:54560) with 4 cores 16/03/14 09:59:54 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20160314095954-0001/0 on hostPort
> ip-11.11.11.11.ec2.internal:54560 with 4  cores, 10.0 GB RAM 16/03/14
> 09:59:54 INFO AppClient$ClientActor: Executor added:
> app-20160314095954-0001/1 on
> worker-20160314095247-ip-11.11.11.11.ec2.internal-46194 (ip
> -11.11.11.11.ec2.internal:46194) with 4 cores 16/03/14 09:59:54 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20160314095954-0001/1 on hostPort
> ip-11.11.11.11.ec2.internal:46194 with 4  cores, 10.0 GB RAM 16/03/14
> 09:59:54 INFO AppClient$ClientActor: Executor added:
> app-20160314095954-0001/2 on
> worker-20160314095247-ip-11.11.11.11.ec2.internal-38895 (ip
> -11.11.11.11.ec2.internal:38895) with 4 cores 16/03/14 09:59:54 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20160314095954-0001/2 on hostPort
> ip-11.11.11.11.ec2.internal:38895 with 4  cores, 10.0 GB RAM 16/03/14
> 09:59:54 INFO AppClient$ClientActor: Executor updated:
> app-20160314095954-0001/1 is now LOADING 16/03/14 09:59:54 INFO
> AppClient$ClientActor: Executor updated: app-20160314095954-0001/0 is
> now LOADING 16/03/14 09:59:54 INFO AppClient$ClientActor: Executor
> updated: app-20160314095954-0001/2 is now LOADING 16/03/14 09:59:54
> INFO AppClient$ClientActor: Executor updated:
> app-20160314095954-0001/0 is now RUNNING 16/03/14 09:59:54 INFO
> AppClient$ClientActor: Executor updated: app-20160314095954-0001/1 is
> now RUNNING 16/03/14 09:59:54 INFO AppClient$ClientActor: Executor
> updated: app-20160314095954-0001/2 is now RUNNING

如何禁用信息日志？

Answer 1

您可以在 JAR 的资源中创建log4j.xml文件并添加以下内容：

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
   <logger name="org.apache.spark">
        <level value="warn"/>
   </logger>
</log4j:configuration>

它将所有org.apache.spark消息的默认记录器级别设置为 WARN