OptionConverter.Hadoop Mapreduce任务convertLevel错误.

Question

我得到一个奇怪的错误，而在我的Hadoop集群执行mapreduce作业。此错误是间歇性的。有时，它会使映射器失败，有时即使抛出错误也会成功。

2022-12-28 01:20:53,882 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org/apache/log4j/helpers/OptionConverter.convertLevel(Ljava/lang/String;Lorg/apache/logging/log4j/Level;)Lorg/apache/logging/log4j/Level; (loaded from file:/opt/hadoop-3.3.0/share/hadoop/common/lib/log4j-1.2.17.jar by sun.misc.Launcher$AppClassLoader@350ed555) called from class org.apache.log4j.config.PropertiesConfiguration (loaded from file:/data/hadoop/yarn/usercache/hdfs-user/appcache/application_1671477750397_6197/filecache/10/job.jar/job.jar by sun.misc.Launcher$AppClassLoader@350ed555).

Gradle文件

plugins {
id 'java'
id "com.github.davidmc24.gradle.plugin.avro" version "1.3.0"
}
dependencies {
compileOnly group: 'org.apache.hadoop', name: 'hadoop-client', version: '3.3.0'
implementation group: 'org.apache.avro', name: 'avro', version: '1.8.1'
implementation group: 'org.apache.avro', name: 'avro-mapred', version: '1.10.2'
implementation group: 'io.netty', name: 'netty-buffer', version: '4.1.51.Final'
implementation  group: 'joda-time', name: 'joda-time', version:'2.8.1'
implementation group: 'org.javatuples', name: 'javatuples', version: '1.2'
implementation group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: '2.12.4'
testImplementation group: 'junit', name: 'junit', version: "$junit4Version"
implementation project(':libraries:java-dcl')
implementation project(':libraries:sdp')
testImplementation group: 'org.assertj', name: 'assertj-core', version: '3.6.1'
}
generateAvroJava {
source("${projectDir}/src/main/avro")//sourcepath avrofile
}
// The alternative to below is to put all java that is dependent on scala (or on the java dependent on the scala) in the src/main/scala dir
// https://stackoverflow.com/questions/23261075/compiling-scala-before-alongside-java-with-gradle
sourceSets {
main {
java {
srcDirs = ['src/main/java']
}
}
}

Answer 1

我必须使用org.apache.log4j。在所有内部库中删除了Logger包，并删除了其他Logger类。有许多文件需要修改。