MapReduce on Cassandra

Question

我正在编写一个简单的MapReduce程序，从Cassandra列族读取数据，但遇到以下错误。任何关于如何进行的提示都是非常感谢的。提前感谢!

Cassandra version : 1.0.3
Hadoop version : 0.20.2
HADOOP_CLASSPATH has: apache-cassandra-1.0.3.jar, libthrift-0.6.jar, commons-lang-2.4.jar and guava-10.0.1.jar
What works : Hadoop MR word count example, Reads from Cassandra column family using cassandra-cli, Thrift and Hector

错误:

11/12/01 20:05:23 INFO mapred.JobClient: Running job: job_201112010835_0009<br/>
11/12/01 20:05:24 INFO mapred.JobClient:  map 0% reduce 0%<br/>
11/12/01 20:05:33 INFO mapred.JobClient: Task Id : attempt_201112010835_0009_m_000000_0, Status : FAILED<br/>
Error: java.lang.ClassNotFoundException: com.google.common.collect.AbstractIterator<br/>
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)<br/>
        at java.security.AccessController.doPrivileged(Native Method)<br/>
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)<br/>
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)<br/>
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)<br/>
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)<br/>
        at java.lang.ClassLoader.defineClass1(Native Method)<br/>
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)<br/>
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)<br/>
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)<br/>
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)<br/>
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)<br/>
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)<br/>
        at java.security.AccessController.doPrivileged(Native Method)<br/>
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)<br/>
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)<br/>
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)<br/>
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)<br/>
        at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.initialize(ColumnFamilyRecordReader.java:158)<br/>
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)<br/>
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)<br/>
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)<br/>
        at org.apache.hadoop.mapred.Child.main(Child.java:170)<br/>

Answer 1

您是否将cassandra库添加到所有任务跟踪器的类路径中?来自维基页面http://wiki.apache.org/cassandra/HadoopSupport:

One configuration note on getting the task trackers to be able to perform queries over
Cassandra: you'll want to update your HADOOP_CLASSPATH in your <hadoop>/conf/hadoop-env.sh 
to include the Cassandra lib libraries. For example you'll want to do something like 
this in the hadoop-env.sh on each of your task trackers:

export HADOOP_CLASSPATH=/opt/cassandra/lib/*:$HADOOP_CLASSPATH

这个示例中的路径显然应该替换为您系统上cassandra库的正确路径。

Answer 2

1)您必须确保用于Hadoop的所有Cassandra jar(所有您在Cassandra二进制安装的lib目录中找到的jar)的版本与Hadoop版本兼容。

2)另外，Cassandra jar可能有一些依赖，你应该有正确版本的依赖jar。

3)由于jar的版本不匹配，我遇到了同样的问题。