Hadoop Mapreduce:从未调用过Reducer

Question

我有一个简单的Map/Reduce任务。Mapper被正常调用和执行，但reducer永远不会被调用。

配置:

conf.setJobName("Index Builder");
conf.setSpeculativeExecution(false);
FileInputFormat.setInputPaths(conf, new Path(args[0].toString()));
FileOutputFormat.setOutputPath(conf, new Path(args[1].toString()));
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(NullOutputFormat.class);
conf.setMapperClass(IndexMapper.class);
conf.setReducerClass(IndexReducer.class);
conf.setMapOutputKeyClass(NullWritable.class);
conf.setMapOutputValueClass(NullWritable.class);
conf.setOutputValueClass(NullWritable.class);
conf.setOutputKeyClass(NullWritable.class);

Mapper签名:

public class IndexMapper extends MapReduceBase implements
  Mapper<LongWritable, Text, NullWritable, NullWritable> {
   @Override
   public void map(LongWritable key, Text val,
     OutputCollector<NullWritable, NullWritable> output,
     Reporter reporter) throws IOException {
        // MAP FUNCTION
   }
}

ٌ减速器签名:

public class IndexReducer extends MapReduceBase implements
  Reducer<NullWritable, NullWritable, NullWritable, NullWritable> {
   @Override
   public void reduce(NullWritable arg0, Iterator<NullWritable> arg1,
     OutputCollector<NullWritable, NullWritable> arg2, Reporter reporter)
     throws IOException {
      // REDUCE CODE
   }
}

Answer 1

您将映射器的输出格式设置为NullOutputFormat，因此它不会生成任何内容。所以不会调用reducer，因为它需要一些数据来运行。