如何提取密钥，使用MapReduce从HBase序列f中值对

Question

我使用Hbase Export utility tool将HBase表导出到HDFS中为SequenceFile。

现在我想使用MapReduce作业来处理此文件：

public class MapSequencefile {
        public static class MyMapper extends Mapper<LongWritable, Text, Text, Text>{
            @Override
            protected void map(LongWritable key, Text value,
                    Mapper<LongWritable, Text, Text, Text>.Context context)
                    throws IOException, InterruptedException {
                System.out.println(key+"...."+value);
            }
        }
        public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {  
            Configuration conf = new Configuration();
            Job job = Job.getInstance(conf , MapSequencefile.class.getSimpleName());
            job.setJarByClass(MapSequencefile.class);
            job.setNumReduceTasks(0);
            job.setMapperClass(MyMapper.class);
            job.setMapOutputKeyClass(Text.class);
            job.setMapOutputValueClass(Text.class);
            job.setInputFormatClass(SequenceFileInputFormat.class); //use SequenceFileInputFormat
            FileInputFormat.setInputPaths(job, "hdfs://192.16.31.10:8020/input/");
            FileOutputFormat.setOutputPath(job, new Path("hdfs://192.16.31.10:8020/out/"));
            job.waitForCompletion(true);
        }  
}

，但它总是抛出以下例外：

Caused by: java.io.IOException: Could not find a deserializer for the Value class: 'org.apache.hadoop.hbase.client.Result'. Please ensure that the configuration 'io.serializations' is properly configured, if you're using custom serialization.
    at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1964)
    at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1811)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1760)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1774)
    at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)

我该怎么办才能解决此错误？

Answer 1

我假设您正在使用它来进行导出：

$ bin/hbase org.apache.hadoop.hbase.mapreduce.Export <tablename> <outputdir> [<versions> [<starttime> [<endtime>]]]

如此HBase页面所述：http：//hbase.apache.org/0.94/book/ops_mgt.html#export

查看org.apache.hadoop.hbase.mapreduce.Export的源代码，您可以看到它设置：

job.setOutputFormatClass(SequenceFileOutputFormat.class);
job.setOutputKeyClass(ImmutableBytesWritable.class);
job.setOutputValueClass(Result.class);

与您的错误一致（值是Result对象）：

Could not find a deserializer for the Value class: 'org.apache.hadoop.hbase.client.Result'

因此您的地图签名需要更改为：

Mapper<ImmutableBytesWritable, Result, Text, Text>

您需要在项目中包含正确的HBase库，因此它可以访问：

org.apache.hadoop.hbase.client.Result