zookeeper在Hadoop 2.2.0上为Giraph 1.0配置



堆叠交换和Giraph的新手,因此请忽略错误并提出任何澄清问题。

操作系统:ubuntu 13.10

Hadoop/Yarn:Hadoop-2.20/(2节点集群)

Giraph:1.0.0(编辑:trunk)

当我尝试运行以下示例时,我得到了一个NullPointerException(NPE):

$hadoop jar$GIRAPH_HOME/GIRAPH examples/target/GIRAPH-examples-1.1.0-SNAPSHOT-for-hadoop-2.20-jar-with-dependences.jarorg.apache.giraph.GiraphRunnerorg.apache.giraph.examples.SimpleShortestPathsComputation-viforg.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertex输入格式-vip/user/hduser/rrdata/tiny_graph.txt-vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat-op/user/hduser/rrdata/output/tiny_graph.out-w 1

堆栈跟踪:

线程"main"java.lang.NullPointerException中的异常org.apache.giraph.syarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.java:460)在org.apache.giraph.syarn.GiraphYarnClient.run(GiraphYarnClient.java:116)网址:org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)org.apache.hoop.util.ToolRunner.run(ToolRunner.java:70)org.apache.hoop.util.ToolRunner.run(ToolRunner.java:84)org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)位于的sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在java.lang.reflect.Method.ioke(Method.java:606)org.apache.hadop.util.RunJar.main(RunJar.java:212)

这似乎与动物园管理员有关。我安装了zookeeper,但之前没有使用过它,看起来配置是错误的。我尝试过-Dgiraph.zkList=hostname:port和相关选项,但出现了"无法识别的选项"异常。

正在为此场景查找正确的动物园管理员设置。如果我想明白了,我会给你回复的。

这是一个如何指定-D标志的示例:

hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2-jar-with-dependencies.jar    org.apache.giraph.GiraphRunner -D giraph.zkList="zkNode.net:2081"  org.apache.giraph.examples.SimpleShortestPathsComputation  -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/rav/giraph/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/rav/giraph/output/shortestpaths -w 1

顺便说一句,Giraph还不支持本地动物园管理员(GiraphYarnClient):

  /**
  * Check if the job's configuration is for a local run. These can all be
  * removed as we expand the functionality of the "pure YARN" Giraph profile.
  */
  private void checkJobLocalZooKeeperSupported() {
     final boolean isZkExternal = giraphConf.isZookeeperExternal();
     final String checkZkList = giraphConf.getZookeeperList();
     if (!isZkExternal || checkZkList.isEmpty()) {
        throw new IllegalArgumentException("Giraph on YARN does not currently" +
            "support Giraph-managed ZK instances: use a standalone ZooKeeper.");
     }
  }

不幸的是,checkZkList为NULL,因此您永远不会看到此异常:)

NPE的原因可能是缺少检查ZK设置的giraphConf。我认为这是由于早期的问题。看起来examples jar不是使用-yj参数提供的。使用"hadoopjar"运行的jar通常是giraph核心本身。

祝你好运,如果你有更多的麻烦,请在Giraph用户列表上发帖。

相关内容

  • 没有找到相关文章

最新更新