MapReduce二次排序不起作用

Question

我正在尝试使用组合键在mapreduce中进行二次排序，该组合键包括：

• 字符串自然键 = 程序名称

• 用于排序的长键 = 自 1970 年以来以毫为单位的时间

问题是排序后我根据整个组合键得到很多化简器

通过调试，我已经验证了哈希代码和比较函数是否正确。从调试日志记录中，每个块来自不同的化简器，它表明分组或分区未成功。从调试日志：

14/12/14 00:55:12 INFO popularitweet.EtanReducer: key=the voice
14/12/14 00:55:12 INFO popularitweet.EtanReducer: the voice: Thu Dec 11 17:51:03 +0000 2014
14/12/14 00:55:12 INFO popularitweet.EtanReducer: the voice: Thu Dec 11 17:51:03 +0000 2014
14/12/14 00:55:12 INFO popularitweet.EtanReducer: key the voice ended

14/12/14 00:55:12 INFO popularitweet.EtanReducer: key=top gear
14/12/14 00:55:12 INFO popularitweet.EtanReducer: top gear: Thu Dec 11 17:51:04 +0000 2014
14/12/14 00:55:12 INFO popularitweet.EtanReducer: key top gear ended

14/12/14 00:55:12 INFO popularitweet.EtanReducer: key=american horror story
14/12/14 00:55:12 INFO popularitweet.EtanReducer: american horror story: Thu Dec 11 17:51:04 +0000 2014
14/12/14 00:55:12 INFO popularitweet.EtanReducer: key american horror story ended

14/12/14 00:55:12 INFO popularitweet.EtanReducer: key=the voice
14/12/14 00:55:12 INFO popularitweet.EtanReducer: the voice: Thu Dec 11 17:51:04 +0000 2014
14/12/14 00:55:12 INFO popularitweet.EtanReducer: key the voice ended

如您所见，语音被发送到两个不同的化简器，但时间戳不同。任何帮助将不胜感激。组合键是以下类：

public class ProgramKey implements WritableComparable<ProgramKey> {
private String program;
private Long timestamp;
public ProgramKey() {
}
public ProgramKey(String program, Long timestamp) {
    this.program = program;
    this.timestamp = timestamp;
}
@Override
public int compareTo(ProgramKey o) {
    int result = program.compareTo(o.program);
    if (result == 0) {
        result = timestamp.compareTo(o.timestamp);
    }
    return result;
}
@Override
public void write(DataOutput dataOutput) throws IOException {
    WritableUtils.writeString(dataOutput, program);
    dataOutput.writeLong(timestamp);
}
@Override
public void readFields(DataInput dataInput) throws IOException {
    program = WritableUtils.readString(dataInput);
    timestamp = dataInput.readLong();
}

我的内置分区器、分组比较器和排序比较器是这些：

public class ProgramKeyPartitioner extends Partitioner<ProgramKey, TweetObject> {
@Override
public int getPartition(ProgramKey programKey, TweetObject tweetObject, int numPartitions) {
    int hash = programKey.getProgram().hashCode();
    int partition = hash % numPartitions;
    return partition;
}

}

public class ProgramKeyGroupingComparator extends WritableComparator {
protected ProgramKeyGroupingComparator() {
    super(ProgramKey.class, true);
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
    ProgramKey k1 = (ProgramKey) a;
    ProgramKey k2 = (ProgramKey) b;
    return k1.getProgram().compareTo(k2.getProgram());
}

}

public class TimeStampComparator extends WritableComparator {
protected TimeStampComparator() {
    super(ProgramKey.class, true);
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
    ProgramKey ts1 = (ProgramKey)a;
    ProgramKey ts2 = (ProgramKey)a;
    int result = ts1.getProgram().compareTo(ts2.getProgram());
    if (result == 0) {
        result = ts1.getTimestamp().compareTo(ts2.getTimestamp());
    }
    return result;
}

}

    public static void main(String[] args) throws IOException,
        InterruptedException, ClassNotFoundException {

    // Create configuration
    Configuration conf = new Configuration();
    // Create job
    Job job = new Job(conf, "test1");
    job.setJarByClass(EtanMapReduce.class);
    // Set partitioner keyComparator and groupComparator
    job.setPartitionerClass(ProgramKeyPartitioner.class);
    job.setGroupingComparatorClass(ProgramKeyGroupingComparator.class);
    job.setSortComparatorClass(TimeStampComparator.class);
    // Setup MapReduce
    job.setMapperClass(EtanMapper.class);
    job.setMapOutputKeyClass(ProgramKey.class);
    job.setMapOutputValueClass(TweetObject.class);
    job.setReducerClass(EtanReducer.class);
    // Specify key / value
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(TweetObject.class);
    // Input
    FileInputFormat.addInputPath(job, inputPath);
    job.setInputFormatClass(TextInputFormat.class);
    // Output
    FileOutputFormat.setOutputPath(job, outputDir);
    job.setOutputFormatClass(TextOutputFormat.class);
    // Delete output if exists
    FileSystem hdfs = FileSystem.get(conf);
    if (hdfs.exists(outputDir))
        hdfs.delete(outputDir, true);
    // Execute job
    logger.info("starting job");
    int code = job.waitForCompletion(true) ? 0 : 1;
    System.exit(code);
}    

Answer 1

编辑...

您的时间戳比较器似乎有一个错别字...您正在将 TS2 设置为 A，而它应该设置为 B：

ProgramKey ts1 = (ProgramKey)a;
ProgramKey ts2 = (ProgramKey)a;

何时应为：

ProgramKey ts1 = (ProgramKey)a;
ProgramKey ts2 = (ProgramKey)b;

这将导致键/值对

排序不正确，并使分组比较器对键/值对排序的假设无效。

还要检查原始程序名称是否为 UTF-8，因为这是 WritableUtils 所假设的。系统的默认代码页是否也是 UTF-8？