已经检查过了，我不明白为什么会遇到此错误。
映射
public class movieMapper extends Mapper<LongWritable, Text, IntWritable, Text> {
public void map(LongWritable key, Text value, Context context ) throws IOException,InterruptedException {
String token[]= value.toString().trim().split("::");
int movieID=Integer.parseInt(token[0].trim());
context.write(new IntWritable(movieID), new Text(token[1].trim()));
}
}
还原剂
public class joinReducer extends Reducer<IntWritable, Text, Text, Text> {
public void reduce(IntWritable key, Iterable<Text> values, Context context) throws IOException,InterruptedException {
float avgRating=0.0f;
int tokenCount = 0;
float ratingSum=0.0f;
int count=0;
String movieName="";
for(Text val:values) {
tokenCount+=1;
}
//If we have more than 40 views/ratings
if(tokenCount-1>40) {
for(Text val:values) {
String temp = val.toString();
if(val.equals("1")||val.equals("2")||val.equals("3")||val.equals("4")||val.equals("5")) {
float tempRating= Float.parseFloat(val.toString().trim());
ratingSum += tempRating;
count++;
}
else {
movieName=val.toString().trim();
}
}
avgRating = ratingSum/ (float)count;
context.write(new Text(Float.toString(avgRating)), new Text(movieName));
}
}
}
驱动程序配置
Configuration conf= new Configuration();
String parameter[]= new GenericOptionsParser(conf,args).getRemainingArgs();
if(parameter.length!=3) {
System.err.println("Three arguments needed <File1> <File2> <Out>");
System.exit(2);
}
//set Driver class
Job job1 = Job.getInstance(conf, "Join");
job1.setJarByClass(MyDriver.class);
job1.setReducerClass(joinReducer.class);
MultipleInputs.addInputPath(job1, new Path(parameter[0]), TextInputFormat.class, movieMapper.class);
MultipleInputs.addInputPath(job1, new Path(parameter[1]), TextInputFormat.class, ratingMapper.class);
job1.setMapOutputKeyClass(IntWritable.class);
job1.setMapOutputValueClass(Text.class);
job1.setOutputKeyClass(Text.class);
job1.setOutputValueClass(Text.class);
FileOutputFormat.setOutputPath(job1, new Path(parameter[2] + "/temp"));
job1.waitForCompletion(true);
13-06-18 09：47：20 信息地图减少。作业：在优步模式下运行的作业job_1528823320386_0018：假 13-06-18 09：47：20 信息地图减少。作业：映射 0% 减少 0% 13-06-18 09：47：24 信息地图减少。作业：任务 ID ： attempt_1528823320386_0018_m_000000_0，状态 ： 失败 错误：java.io.IOException：在映射中的键中键入不匹配：预期的org.apache.hadoop.io.IntWritable，收到org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java：1069( at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java：712( at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java：89( at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java：112( at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java：124( at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java：145( at org.apache.hadoop.mapreduce.lib.input.DelegatingMapper.run(DelegatingMapper.java：55( at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java：784( at org.apache.hadoop.mapred.MapTask.run(MapTask.java：341( at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java：168( at java.security.AccessController.doPrivileged(Native Method( at javax.security.auth.Subject.doAs(Subject.java：422( at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java：1642( at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java：163(
13-06-18 09：47：25 信息地图减少。作业：地图 50% 减少 0% 13-06-18 09：47：29 信息地图减少。作业：任务 ID ： attempt_1528823320386_0018_m_000000_1，状态 ： 失败 错误：java.io.IOException：在映射中的键中键入不匹配：预期的org.apache.hadoop.io.IntWritable，收到org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java：1069( at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java：712( at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java：89( at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java：112( at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java：124( at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java：145( at org.apache.hadoop.mapreduce.lib.input.DelegatingMapper.run(DelegatingMapper.java：55( at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java：784( at org.apache.hadoop.mapred.MapTask.run(MapTask.java：341( at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java：168( at java.security.AccessController.doPrivileged(Native Method( at javax.security.auth.Subject.doAs(Subject.java：422( at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java：1642( at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java：163(
此作业中运行两个映射器，movieMapper 和 ratingMapper。ratingMapper 在函数声明中有一个拼写错误的关键字，并且 map 函数的名称 'map' 被错误地写成 'reduce'。
根据 out 配置，化简器应该接受 IntWriable 类型的密钥，但正在获得 LongWwriteable，因此出现了错误。(文本输入格式生成 LongWwrite 类型的键和文本类型的值(