使用Log4j在日志中输出Spark应用程序id

Question

我有一个用于Spark应用程序的自定义Log4j文件。我想输出Spark应用程序id以及其他属性，如消息和日期，因此JSON字符串结构如下所示：

{"name":,"time":,"date":,"level":,"thread":,"message":,"app_id":}

现在，这个结构看起来是这样的：

{"name":,"time":,"date":,"level":,"thread":,"message":}

如何为Spark驱动程序日志定义这样的布局？

我的log4j文件如下：

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j='http://jakarta.apache.org/log4j/'>
<appender name="Json" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.hadoop.log.Log4Json">
<param name="ConversionLayout" value=""/>
</layout>
</appender>
<root>
<level value="INFO"/>
<appender-ref ref="Json"/>
</root>
</log4j:configuration>

Answer 1

我怀疑org.apache.hadoop.log.Log4Json是否可以为此目的进行调整。根据它的javadoc和源代码，它可能相当麻烦。

虽然看起来您正在使用Log4j1x，但其API非常灵活，我们可以通过扩展org.apache.log4j.Layout轻松定义自己的布局。

我们需要一个case类，它将根据目标结构转换为JSON：

case class LoggedMessage(name: String,
appId: String,
thread: String,
time: Long,
level: String,
message: String)

并且CCD_ 3可以如下扩展。要访问"app_id"的值，我们将使用Log4j的映射诊断上下文

import org.apache.log4j.Layout
import org.apache.log4j.spi.LoggingEvent
import org.json4s.DefaultFormats
import org.json4s.native.Serialization.write
class JsonLoggingLayout extends Layout {
// required by the API
override def ignoresThrowable(): Boolean = false
// required by the API
override def activateOptions(): Unit = { /* nothing */ }
override def format(event: LoggingEvent): String = {
// we are using json4s for JSON serialization
implicit val formats = DefaultFormats
// retrieve app_id from Mapped Diagnostic Context
val appId = event.getMDC("app_id") match {
case null => "[no_app]" // logged messages outside our app
case defined: AnyRef => defined.toString
}
val message = LoggedMessage("TODO",
appId,
Thread.currentThread().getName,
event.getTimeStamp,
event.getLevel.toString,
event.getMessage.toString)
write(message) + "n"
}
}

最后，在创建Spark会话时，我们将app_id值放入MDC:

import org.apache.log4j.{Logger, MDC}
// create Spark session
MDC.put("app_id", session.sparkContext.applicationId)
logger.info("-------- this is info --------")
logger.warn("-------- THIS IS A WARNING --------")
logger.error("-------- !!! ERROR !!! --------")

这会生成以下日志：

{"name":"TODO","appId":"local-1550247707920","thread":"main","time":1550247708149,"level":"INFO","message":"-------- this is info --------"}
{"name":"TODO","appId":"local-1550247707920","thread":"main","time":1550247708150,"level":"WARN","message":"-------- THIS IS A WARNING --------"}
{"name":"TODO","appId":"local-1550247707920","thread":"main","time":1550247708150,"level":"ERROR","message":"-------- !!! ERROR !!! --------"}

当然，不要忘记参考log4j-config-xml:中的实现

<appender name="Json" class="org.apache.log4j.ConsoleAppender">
<layout class="stackoverflow.q54706582.JsonLoggingLayout" />
</appender>