可写可比较对象不可序列化

Question

我有以下MR作业类，但是当我运行作业时，作业失败，出现以下异常，敬请建议。

    public class MongoKey implements WritableComparable<MongoKey> {
    ...
    private Text name;
private Text place;
public MongoKey() {
    this.name = new Text();
    this.place = new Text();
}
public MongoKey(Text name, Text place) {
    this.name = name;
    this.place = place;
}
public void readFields(DataInput in) throws IOException {
    name.readFields(in);
    place.readFields(in);
}
public void write(DataOutput out) throws IOException {
    name.write(out);
    place.write(out);
}
    public int compareTo(MongoKey o) {
            MongoKey other = (MongoKey)o;
            int cmp = name.compareTo(other.name);
            if(cmp != 0){
                return cmp;
            }
            return place.compareTo(other.place);
        }
    }
public class MongoValue implements Writable {
...
public void readFields(DataInput in) throws IOException {
        profession.readFields(in);
    }
    public void write(DataOutput out) throws IOException {
        profession.write(out);
    }
}
public class MongoReducer extends Reducer<MongoKey, MongoValue, MongoKey, BSONWritable> {
...
context.write(key, new BSONWritable(output)); // line 41
}
public class MongoHadoopJobRunner extends Configured implements Tool {
    public int run(String[] args) throws Exception {
        if (args.length != 2) {
            System.out.println("usage: [input] [output]");
            System.exit(-1);
        }
        Configuration conf = getConf();
        for (String arg : args)
            System.out.println(arg);
        GenericOptionsParser parser = new GenericOptionsParser(conf, args);
        conf.set("mongo.output.uri", "mongodb://localhost/demo.logs_aggregate");
        MongoConfigUtil.setOutputURI(conf, "mongodb://localhost/demo.logs_aggregate");
        MongoConfigUtil.setOutputFormat(conf, MongoOutputFormat.class);
        final Job job = new Job(conf, "mongo_hadoop");
        job.setOutputFormatClass(MongoOutputFormat.class);
        // Job job = new Job();
        job.setJarByClass(MongoHadoopJobRunner.class);
        // job.setJobName("mongo_hadoop");
        job.setNumReduceTasks(1);
        job.setMapperClass(MongoMapper.class);
        job.setReducerClass(MongoReducer.class);
        job.setMapOutputKeyClass(MongoKey.class);
        job.setMapOutputValueClass(MongoValue.class);
        job.setOutputKeyClass(MongoKey.class);
        job.setOutputValueClass(BSONWritable.class);
        job.setInputFormatClass(MongoInputFormat.class);
        for (String arg2 : parser.getRemainingArgs()) {
            System.out.println("remaining: " + arg2);
        }
        Path inPath = new Path(parser.getRemainingArgs()[0]);
        MongoInputFormat.addInputPath(job, inPath);
        job.waitForCompletion(true);
        return 0;
    }
    public static void main(String[] pArgs) throws Exception {
        Configuration conf = new Configuration();
        for (String arg : pArgs) {
            System.out.println(arg);
        }
        GenericOptionsParser parser = new GenericOptionsParser(conf, pArgs);
        for (String arg2 : parser.getRemainingArgs()) {
            System.out.println("ree" + arg2);
        }
        System.exit(ToolRunner.run(conf, new MongoHadoopJobRunner(), parser
                .getRemainingArgs()));
    }
}

以下情况除外

java.lang.Exception: java.lang.IllegalArgumentException: can't serialize class com.name.custom.MongoKey
...
...
at com.mongodb.hadoop.output.MongoRecordWriter.write(MongoRecordWriter.java:93)
     at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
     at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
     at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
     at com.name.custom.MongoReducer.reduce(MongoReducer.java:41)
     at com.name.custom.MongoReducer.reduce(MongoReducer.java:11)

代码似乎应该没有任何问题，但是为什么它无法序列化字段，我完全不知道。

提前非常感谢

Answer 1

正如我从源代码中看到MongoRecordWriter它不支持任意WritableComparable对象作为键。您可以使用这些类之一作为键：BSONWritable、BSONObject、Text、UTF8、简单的包装器，如 IntWritable。另外，我认为您可以使用Serializable对象作为键。所以我可以建议你两个解决方法：

1. 使MongoKey可序列化（implements Serializable、实现writeObject、readObject方法）。
2. 使用其中一个受支持的类作为键，例如，您可以使用Text作为键：Text key = new Text(name.toString() + "t" + place.toString());

Answer 2

这个：

java.lang.Exception: java.lang.IllegalArgumentException: can't serialize class com.name.custom.MongoKey

引发异常是因为MongoKey没有实现java.io.Serializable。

将可序列化添加到类声明中