引起原因:java.lang.ClassNotFoundException:org.apache.hadoop.mapr



我正在MAC OSX 10.9.4中进行sqoop导入,并得到错误:

14/10/24 11:51:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
14/10/24 11:51:41 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
14/10/24 11:51:41 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
14/10/24 11:51:41 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
14/10/24 11:51:41 INFO tool.CodeGenTool: Beginning code generation
14/10/24 11:51:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test_log` AS t LIMIT 1
14/10/24 11:51:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test_log` AS t LIMIT 1
14/10/24 11:51:42 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/hadoop/hadoop-2.5.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/2398ed5be0489f1c76d6fb556f3e0e9d/test_log.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
14/10/24 11:51:45 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/2398ed5be0489f1c76d6fb556f3e0e9d/test_log.jar
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/InputFormat
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:51)
    at com.cloudera.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:33)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:483)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.InputFormat
    at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 58 more

Hadoop版本:2.5.1Sqoop版本:1.4.5os:OSX10.9.4

设置env变量:

export HADOOP_COMMON_HOME=/Users/HADOOP/hadop-2.5.1/

export HADOOP_MAPRED_HOME=/Users/HADOOP/hadop-2.5.1/share/HADOOP/mapreduce/

Sqoop jar(Sqoop-1.4.5.jar)位于/Users/hadoop/hadop-2.5.1/lib中/和/Users/hadoop/scoop/lib 中的mysql连接器

中也存在core.jar

/Users/hadoop/hadoop2.5.1/share/hadoop/marepeduce/hadoop-mapreduce-client-core-2.5.1.jar

for f in ${HADOOP_MAPRED_HOME}/*.jar; do
HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:$f;
done  

将其添加到bin/configure sqoop

不知何故,Jars无法访问。在将Sqoop罐子放入HDFS后,Sqoops导入顺利。此链接有帮助找不到Sqoop jar文件

cd$SQOOP_HOME/bin

vim配置sqoop

在线下添加

add_to_classpath ${HADOOP_MAPRED_HOME}

在"add_to_classpath${SQOOP_JAR_DIR}"之后

尝试检查$HADOOP_COMMON_LIB_NATIVE_DIR中是否可以找到库。如果没有,你可以把它放在你的mapreduce实现的同一目录中:

/Users/hadoop/hadoop2.5.1/share/hadoop/marepeduce/

我也遇到了同样的问题!只需将hadoop-mapreduce-client-core-3.0.0-SNAPSHOT.jar从sqooq复制到lib文件夹中。脏,但它工作。您可能只是想进行一些测试

最新更新