如何在Ubuntu上保留两个Hadoop版本

Question

我正在运行一些测试，我需要能够在Hadoop的两个不同版本之间来回切换-2.2.0版本和我自定义的版本。然而，我在~/.bashrc设置它们的变量时遇到了问题-如果我同时拥有这两个版本，就会发生冲突。我只能运行2.2.0版本，而不能运行我的版本。但是，如果我在~/.bashrc中注释与2.2.0版本相关的变量，它将运行我的版本。以下是~/.bashrc:中我的Hadoop相关变量

#Hadoop variables - my version 
export HADOOPTWO_INSTALL=/home/parallels/Development/Hadoop/hadoop-trunk/trunk2/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT
export PATH=$PATH:$HADOOPTWO_INSTALL/bin
export PATH=$PATH:$HADOOPTWO_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOPTWO_INSTALL
export HADOOP_COMMON_HOME=$HADOOPTWO_INSTALL
export HADOOP_HDFS_HOME=$HADOOPTWO_INSTALL
export YARN_HOME=$HADOOPTWO_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOPTWO_INSTALL/lib/native
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOPTWO_INSTALL/lib/"

#Hadoop variables - version 2.2.0
export HADOOP_INSTALL=/home/parallels/Development/Hadoop/hadoop-2.2.0
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_INSTALL/lib/"

如果我运行$HADOOPTWO_INSTALL/bin/hadoop version，它调用我版本的hadoop文件，它将返回：

Hadoop 2.2.0
Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
Compiled by hortonmu on 2013-10-07T06:28Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4
This command was run using /home/parallels/Development/Hadoop/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar

考虑到命令中调用的jar的位置，这显然是错误的(它仍然从2.2.0版本调用jar，尽管我专门调用了我的定制Hadoop版本(。

我做错了什么？如有任何帮助，我们将不胜感激。

Answer 1

在类似的情况下，当我必须拥有多个冲突的环境时，我所做的就是将它们封装到函数中：

function my-hadoop-env {
    #Hadoop variables - my version 
    export HADOOPTWO_INSTALL=/home/parallels/Development/Hadoop/hadoop-trunk/trunk2/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT
    export PATH=$PATH:$HADOOPTWO_INSTALL/bin
    export PATH=$PATH:$HADOOPTWO_INSTALL/sbin
    export HADOOP_MAPRED_HOME=$HADOOPTWO_INSTALL
    export HADOOP_COMMON_HOME=$HADOOPTWO_INSTALL
    export HADOOP_HDFS_HOME=$HADOOPTWO_INSTALL
    export YARN_HOME=$HADOOPTWO_INSTALL
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOPTWO_INSTALL/lib/native
    export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOPTWO_INSTALL/lib/"
}
function hadoop-env {
    #Hadoop variables - version 2.2.0
    export HADOOP_INSTALL=/home/parallels/Development/Hadoop/hadoop-2.2.0
    export PATH=$PATH:$HADOOP_INSTALL/bin
    export PATH=$PATH:$HADOOP_INSTALL/sbin
    export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
    export HADOOP_COMMON_HOME=$HADOOP_INSTALL
    export HADOOP_HDFS_HOME=$HADOOP_INSTALL
    export YARN_HOME=$HADOOP_INSTALL
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
    export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_INSTALL/lib/"
}

然后，当打开一个shell时，执行适当的my-hadoop-env或hadoop-env函数来拉入该shell中所需的环境。

如果你觉得自己很狡猾，你甚至可以让他们修改PS1，这样就可以清楚地知道你在哪一个。

Answer 2

我认为您正在用hadoop 2.2.0路径覆盖HADOOPTWO路径变量。你只能导出1个PATH变量，如果你的.bashrc按照上面发布的顺序执行，那么你的路径应该像这个

/home/parallels/Development/Hadoop/hadoop-2.2.0/bin

Answer 3

我在版本之间切换所做的是将不同的hadoop版本放在不同的目录中，并创建一个到我想要激活的目录的符号链接。这样，所有的配置总是引用相同的位置。

ln -s /usr/local/hadoop-versions/hadoop-2.2.0 /usr/local/hadoop