我们目前正在使用以下log4j jar
log4j-core-2.6.2，log4j-flume-ng-2.6.2
<Flume name="aggregatorApp" compress="false" type="Avro"
ignoreExceptions="true" batchSize="10" blocking="false" >
<Agent host="${flumeHostPrimary}" port="${flumePortPrimary}" />
<Agent host="${flumeHostSecondary}" port="${flumePortSecondary}" />
<FlumeEventFactory logType="APPLICATION"/>
<PatternLayout header="app" pattern="[%-5p] %d %c %X{correlationId} - %m" />
<AsyncLogger name="com.xyz.abc" level="debug"
additivity="false" blocking="false">
<AppenderRef ref="IDRESTSERVICELOG" />
<AppenderRef ref="aggregatorApp" level="INFO" />
<AppenderRef ref="CONSOLE" level="INFO"/>
</AsyncLogger>
- 当flume代理启动并运行应用程序时，日志事件将成功发布到flume代理
- 如果flume代理关闭了一段时间，则应用程序线程会因以下异常而被卡住
；
Exception in thread "elasticsearch[_client_][generic][T#3]" java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-02-01 07:43:23,618 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.apache.catalina.valves.StuckThreadDetectionValve- Thread "http-apr-8080-exec-7" (id=167) has been active for 20,392 milliseconds (since 2/1/19 7:42 AM) to serve the same request for htpps://xyz//directories/v1.0/search and may be stuck (configured threshold for this StuckThreadDetectionValve is 20 seconds). There is/are 1 thread(s) in total that are monitored by this Valve and may be stuck.
2019-02-01 07:46:18,619 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] ERROR org.apache.catalina.core.ContainerBase- Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-apr-8080-exec-42" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-apr-8080-exec-50" Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" Exception in thread "I/O dispatcher 23"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
Exception in thread "http-apr-8080-exec-38" Exception in thread "http-apr-8080-exec-46" Exception in thread "http-apr-8080-exec-53" Exception in thread "I/O dispatcher 17"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-apr-8080-exec-31"
以上异常发生在重负载期间(每个请求1MB日志记录)注意：我们正在进行负载测试
由于应用程序的日志记录量很大，但Flume appender无法以相同的速度使用。我们怀疑Log4j环形缓冲区已满，这会阻塞应用程序线程并造成内存不足的异常。
考虑到Log4J appenders、存在问题，我们已经尝试了一些操作
-Dlog4j2.AncQueueFullPolicy=放弃-Dlog4j2.enable.threadlocals=true-Dlog4j2.enable.direct.encoders=true
但上述系统属性毫无用处。我们是不是错过了什么？
我同意AsyncLogger的RingBuffer会导致内存不足。环形缓冲区的默认大小为256K个条目。JVM的堆大小是多少？
另一种方法是使用FlumeAppender的嵌入变体或持久变体。这些将在转发到Flume代理之前将事件缓存在本地文件通道中。这将允许您通过调整可用磁盘空间来控制服务器可以容忍停机的时间。您仍然可以获得异步记录器的优势，但可以避免停机。