我已经设置了一个在64位linux上运行Hadoop 2.1测试版的集群。然而,每次我运行hadoop命令工具时,都会弹出一条警告消息:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform...
using builtin-java classes where applicable
然后我发现它缺少用于64位linux的本地库。官方的hadoop2.1 tarball只在/lib/native
文件夹中提供了32位linux的原生库。
我阅读了hadoop原生库的官方文档,指南上写着:
Once you installed the prerequisite packages use the standard hadoop
build.xml file and pass along the compile.native flag (set to true) to
build the native hadoop library:
$ant -Dcompile.native=true <target>
我搜索hadoop文件夹,没有名为build.xml
的文件。没有足够的java编程和hadoop知识,所以我想知道如何为64位linux系统编译本地库?谢谢
构建系统已更改为maven。您可以在此处找到建筑说明:https://svn.apache.org/repos/asf/hadoop/common/trunk/BUILDING.txt
具体来说,您可以运行以下操作:mvn包-Pdist,本机,文档-DskipTests-Dtar
(一旦安装了protobuf 2.5.0)
下载并安装protobuf
wget http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz /tmp
tar -xvf protobuf-2.5.0.tar.gz
cd /tmp/protobuf-2.5.0
sudo ./configure
sudo ./make install
sudo ldconfig
安装cmake
sudo apt-get install cmake
使用maven 构建本地库
mvn compile -Pnative
如果有任何错误,使用-e -X
开关运行maven,它将输出详细的调试信息。查看错误是什么,并进行适当的更改。
例如,我得到了以下错误:
[ERROR] Could not find goal 'protoc' in plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT among available goals -> [Help 1]
意味着您的protobuf版本不正确。
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/tmp/hadoop-2.5.0-src/hadoop-common-project/hadoop-common/target/native"): error=2, No such file or directory
cmake未安装。
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while creating archive:
[ERROR] Exit code: 1 - /tmp/hadoop-2.5.0-src/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java:27: error: unexpected end tag: </ul>
[ERROR] * </ul>
不知道问题出在哪里。只需通过将-Dmaven.javadoc.skip=true
标志传递给maven来跳过javadoc生成。