为什么" set"只有一个元素,例如,它应该具有4个具有相同URL和4个不同IP的输入的元素。我还使用了" for-east",而不是"迭代器",但不起作用。有人可以帮我吗?
mapper
public class WordCount {
public static class TokenizerMapper extends Mapper<Object, Text, Text, Text> {
private Text IP = new Text();
private Text word = new Text();
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
String[] tokens = line.split(",");
word.set(tokens[2]);
IP.set(tokens[0]);
context.write(word, IP);
}
}
还原器
public static class IntSumReducer extends Reducer<Text, Text, Text, Text> {
public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
Set<String> set = new HashSet<String>();
Iterator<Text> iterator = values.iterator();
while (iterator.hasNext()) {
set.add(iterator.next().toString());
}
int a = set.size();
String str = String.format("%d", a);
context.write(key, new Text(str));
}
}
作业
public static void main(String[] args) throws Exception {
Job job = new Job();
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
输入
"10.131.0.1","[29/Nov/2017:14:31:33","GET / HTTP/1.1","200"
"10.131.0.2","[29/Nov/2017:14:31:38","GET / HTTP/1.1","200"
"10.131.0.3","[29/Nov/2017:14:31:56","GET / HTTP/1.1","200"
"10.131.0.4","[29/Nov/2017:14:32:02","GET / HTTP/1.1","404"
"10.131.0.5","[29/Nov/2017:16:31:39","GET / HTTP/1.1","200"
"10.131.0.1","[29/Nov/2017:14:05:35","GET /contest.php HTTP/1.1","200"
"10.131.0.2","[29/Nov/2017:14:05:38","GET /contest.php HTTP/1.1","200"
"10.131.0.3","[29/Nov/2017:14:05:50","GET /contest.php HTTP/1.1","404"
"10.131.0.1","[29/Nov/2017:13:51:41","GET /login.php HTTP/1.1","200"
"10.131.0.2","[29/Nov/2017:13:51:49","GET /login.php HTTP/1.1","200"
"10.131.0.1","[29/Nov/2017:13:51:46","GET /contestproblem.php?name=RUET%20OJ%20Server%20Testing%20Contest HTTP/1.1","200"
"10.131.0.8","[29/Nov/2017:13:51:46","GET /contestproblen.php?name=RUET%20OJ%20Server%20Testing%20Contest HTTP/1.1","200"
我的结果是
"GET / HTTP/1.1" 1
"GET /contest.php HTTP/1.1" 1
"GET /contestproblem.php?name=RUET%20OJ%20Server%20Testing%20Contest HTTP/1.1" 1
"GET /contestproblen.php?name=RUET%20OJ%20Server%20Testing%20Contest HTTP/1.1" 1
"GET /login.php HTTP/1.1" 1
还原器工作正常,但是组合仪没有做您想的事情。Combiner打开的情况是:
映射器输出:
("GET / HTTP/1.1", "10.31.0.1")
("GET / HTTP/1.1", "10.31.0.2")
组合仪输入:
("GET / HTTP/1.1", {"10.31.0.1", "10.31.0.2"})
组合器输出:
("GET / HTTP/1.1", "2") //You have the right answer here...
还原器输入:
("GET / HTTP/1.1", {"2"}) //...but then it gets passed into the Reducer again
还原器输出:
("GET / HTTP/1.1", "1")
只有一个元素进入还原器,因此将其减少到" 1"。
删除组合仪(删除job.setCombinerClass(IntSumReducer.class);
,这将起作用。
其他推荐更改:
- 具有还原器输出
IntWritable
,而不是将一个数字转换为Text
。 - 为了节省昂贵的
Text -> String
转换,使Set
代替Set<Text>
而不是Set<String>
。