我有一组只需要处理男性记录的记录,在map reduce程序中,我使用if条件只过滤男性记录。但下面的程序给出了零记录作为输出。
输入文件:
1,布兰登·巴克纳,阿维尔,女,525
2,韦达·霍普金斯,阿维尔,男,633
3,Zia Underwood,扑热息痛,男性,980
4,Austin Mayer,扑热息痛,女性,338
5,马拉·希金斯,阿维尔,女,153
6,Sybill Crosby,阿维尔,男,193
7,Tyler Rosales,扑热息痛,男性,778
8,Ivan Hale,阿维尔,女,454
9,Alika Gilmore,扑热息痛,女性,833
10,Len Burgess,metacin,男性,325
Mapreduce程序:
package org.samples.mapreduce.training;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public class patientrxMR_filter {
public static class MapDemohadoop extends
Mapper<LongWritable, Text, Text, IntWritable> {
// setup , map, run, cleanup
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
String[] elements = line.split(",");
String gender =elements[3];
if ( gender == "male" ) {
Text tx = new Text(elements[2]);
int i = Integer.parseInt(elements[4]);
IntWritable it = new IntWritable(i);
context.write(tx, it);
}
}
}
public static class Reduce extends
Reducer<Text, IntWritable, Text, IntWritable> {
// setup, reduce, run, cleanup
// innput - para [150,100]
public void reduce(Text key, Iterable<IntWritable> values,
Context context) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
if (args.length != 2) {
System.err.println("Insufficient args");
System.exit(-1);
}
Configuration conf = new Configuration();
//conf.set("fs.default.name","hdfs://localhost:50000");
conf.set("mapred.job.tracker", "hdfs://localhost:50001");
// conf.set("DrugName", args[3]);
Job job = new Job(conf, "Drug Amount Spent");
job.setJarByClass(patientrxMR_filter.class); // class conmtains mapper and
// reducer class
job.setMapOutputKeyClass(Text.class); // map output key class
job.setMapOutputValueClass(IntWritable.class);// map output value class
job.setOutputKeyClass(Text.class); // output key type in reducer
job.setOutputValueClass(IntWritable.class);// output value type in
// reducer
job.setMapperClass(MapDemohadoop.class);
job.setReducerClass(Reduce.class);
job.setNumReduceTasks(1);
job.setInputFormatClass(TextInputFormat.class); // default -- inputkey
// type -- longwritable
// : valuetype is text
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}
if ( gender == "male" )
这行不适用于相等性检查,对于java中的相等性,请使用object.equals((
i.e
if ( gender.equals("male") )
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
String[] elements = line.split(",");
Hadoop使用分布式文件系统,在"String-line=value.toString((;"行是块中具有偏移量(键(的文件内容。在这种情况下,该行加载整个测试文件,该文件显然可以放入一个块中,而不是文件中的每一行。