如何按值或计数对单词计数程序进行排序

Question

如何按计数/值而不是按键对wordcount输出进行排序。

在正常情况下，输出为

hi 2
hw 3 
wr 1 
r 3

但是期望的输出是

wr 1
hi 2
hw 3
r 3

我的代码是：

public class sortingprog {
     public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, IntWritable, Text> {
         private final static IntWritable one = new IntWritable(1);
         private Text word = new Text();
         public void map(LongWritable key, Text value, OutputCollector<IntWritable, Text> output, Reporter reporter) throws IOException {
           String line = value.toString();
           StringTokenizer tokenizer = new StringTokenizer(line);
           while (tokenizer.hasMoreTokens()) {
             word.set(tokenizer.nextToken());
             output.collect(one,word);
           }
         }
       }

     public static class Reduce extends MapReduceBase implements Reducer<IntWritable,Text, IntWritable, Text> {
     public void reduce(Iterator<IntWritable> key, Text value, OutputCollector<IntWritable, Text> output, Reporter reporter) throws IOException {
            int sum=0;
           while (key.hasNext()) {
             sum+=key.next().get();
           }
           output.collect(new IntWritable(sum),value);
     }
    @Override
    public void reduce(IntWritable arg0, Iterator<Text> arg1,
            OutputCollector<IntWritable, Text> arg2, Reporter arg3)
            throws IOException {
        // TODO Auto-generated method stub
    }
     }
     public static class GroupComparator extends WritableComparator {
            protected GroupComparator() {
                super(IntWritable.class, true);
            }
            @SuppressWarnings("rawtypes")
            @Override
            public int compare(WritableComparable w1, WritableComparable w2) {
                IntWritable v1 = (IntWritable) w1;
                IntWritable v2 = (IntWritable) w2;          
                return -1 * v1.compareTo(v2);
            }
        }
       public static void main(String[] args) throws Exception {
         JobConf conf = new JobConf(sortingprog.class);
         conf.setJobName("wordcount");

         conf.setOutputKeyClass(IntWritable.class);
         conf.setOutputValueClass(Text.class);

         conf.setMapperClass(Map.class);
         conf.setReducerClass(Reduce.class);
         conf.setOutputValueGroupingComparator(GroupComparator.class);
         conf.setInputFormat(TextInputFormat.class);
         conf.setOutputFormat(TextOutputFormat.class);
         FileInputFormat.setInputPaths(conf, new Path(args[0]));
         FileOutputFormat.setOutputPath(conf, new Path(args[1]));
         JobClient.runJob(conf);
       }
}

Answer 1

您要查找的内容称为"辅助排序"。在这里，您可以找到两个关于如何在MapReduce:中实现短值的教程

http://vangjee.wordpress.com/2012/03/20/secondary-sorting-aka-sorting-values-in-hadoops-mapreduce-programming-paradigm/

http://codingjunkie.net/secondary-sort/

Answer 2

您需要执行以下操作。

1. 创建一个使用这两个字段的自定义可写可比项
2. 在compareTo方法中，提供了比较自定义可写的实现逻辑。减速器稍后会调用此操作来对关键帧进行排序。这是整个实施的关键。在compareTo中，只使用第二个字段来比较值

public CustomPair implements WritableComparable{ public CustomPair(String fld1,int fld2){ this.fld1=fld1; //wr this.fld2=fld2;//1 } @Override public int compareTo(Object o2) { CustomPair other = (CustomPair ) o2; int compareValue = other.fld2().compareTo(this.fld2()); return compareValue; } public void write(DataOutput out) throws IOException { dataOutput.writeUTF(fld1); dataOutput.writeInt(fld2); } // You have to implement the rest of the methods.
}如果你需要额外的帮助，请告诉我。