我想使用 log4j kafka appender 将我的应用程序日志消息流式传输到 kafka。
我的应用程序正在使用 spring boot 和 log4j2。
我在主题中看到启动日志消息,但没有看到应用程序中使用的日志语句。
我在 log4j2 配置中遗漏了什么吗?
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="debug" name="kafka-logs"
packages="myapp.kafka.test">
<Appenders>
<Kafka name="Kafka" topic="test-kafka-topic">
<JSONLayout />
<Property name="bootstrap.servers">server1,server2</Property>
</Kafka>
<Async name="Async">
<AppenderRef ref="Kafka"/>
</Async>
<Console name="stdout" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} %-5p [%-7t] %F:%L - %m%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="Kafka"/>
<AppenderRef ref="stdout"/>
</Root>
<Logger name="org.apache.kafka" level="WARN" />
</Loggers>
绒球.xml
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.11.2</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.11.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.10</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-log4j-appender</artifactId>
<version>1.0.0</version>
<exclusions>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
爪哇代码
@SpringBootApplication
public class TestApplication {
private static final Logger logger =
LoggerFactory.getLogger(TestApplication.class);
public static void main(String[] args) {
SpringApplication application = new SpringApplication(TestApplication.class);
logger.info("Before Application Startup");
application.run(args);
logger.info("After Application Startup");
}
}
卡夫卡日志:
{
"thread" : "main",
"level" : "INFO",
"loggerName" : "TestApplication",
"message" : "Started TestApplication in 17.19 seconds (JVM running for 22.67)",
"endOfBatch" : false,
"loggerFqcn" : "org.apache.commons.logging.LogFactory$Log4jLog",
"instant" : {
"epochSecond" : 1555422668,
"nanoOfSecond" : 856000000
},
"threadId" : 1,
"threadPriority" : 5
}
控制台输出:
08:51:08.856 [main] INFO TestApplication - Started TestApplication in 17.19 seconds (JVM running for 22.67)
[main] INFO TestApplication - Testing my application from TestApplication
[main] ERROR TestApplication - Testing my application from TestApplication
您需要在pom.xml文件中删除春季启动应用程序中日志的默认提供程序,哪些是Logback,也许是Log-classic,然后您需要添加log4j2作为新的日志提供程序并添加Kafka追加器。添加依赖项后,您需要 xml 配置文件,您可以在其中添加 Kafka 追加器配置。默认情况下,您需要在项目的资源路径中找到配置文件,并将其命名为"log4j2.xml"。
你可以找到许多其他的 Log4j2 追加器,如 Cassandra 或故障转移追加器,并将它们添加到配置文件中的 Kafka 追加器旁边。您可以在下面找到一个适用且正确的示例。
<!--excluding logback -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
</exclusions>
</dependency>
<!--adding log4j2 and kafka appender-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-log4j-appender</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
Kafka appender configuration
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="info" name="kafka-appender" packages="Citydi.ElasticDemo">
<Appenders>
<Kafka name="kafkaLogAppender" topic="Second-Topic">
<JSONLayout />
<Property name="bootstrap.servers">localhost:9092</Property>
<MarkerFilter marker="Recorder" onMatch="DENY" onMismatch="ACCEPT"/>
</Kafka>
<Console name="stdout" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} stdout %highlight{%-5p} [%-7t] %F:%L - %m%n"/>
<MarkerFilter marker="Recorder" onMatch="DENY" onMismatch="ACCEPT"/>
</Console>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="kafkaLogAppender"/>
<AppenderRef ref="stdout"/>
</Root>
<Logger name="org.apache.kafka" level="warn" />
</Loggers>
</Configuration>
我认为log4j2已经有了Kafka追加器。至于我,我在 gradle 中添加了 Kafka 客户端。所以无需添加kafka-log4j-appender
compile group: 'org.apache.kafka', name: 'kafka-clients', version: '2.0.1'
Maven应该类似
您需要排除 spring-boot-starter-logging
如下<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
</exclusions>
</dependency>
试试这个:
<appender name="KAFKA" class="org.apache.kafka.log4jappender.KafkaLog4jAppender">
<param name="Topic" value="logs"/>
<param name="BrokerList" value="localhost:9092"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n"/>
</layout>
</appender>
<logger name="org.apache.kafka" additivity="false">
<level value="INFO"/>
<appender-ref ref="KAFKA"/>
</logger>