如何将映射器键作为文本输出到Hadoop中的Reducer

Question

我正在尝试将字符串转换为文本对象，并将值作为MapOutputKey输出到Reducer。但是，在化简器中，键看起来像一个黑色空间，其中值（不是 MapOutputValue）作为内存位置。但是，如果我将字符串硬编码到文本对象中并将其发送出去，我能够在 Reducer 中看到正确的值。编辑：- 现在这非常奇怪和有趣。相同的程序适用于较小的数据集，即从映射器发送的文本密钥在化简器中接收。但是如果我使用原始数据集，则从map发送的文本键不会反映在化简器中。我不知道，关于Hadoop，有些事情我不知道。这是完整的源代码：-

 public class InvertedIndex {
    public InvertedIndex(){}
    public static class InvertedMap extends Mapper<IntWritable,Text,Text,DocAttributes> {
        Integer docNum;
        String word;
        Integer docFrequency;
        Integer termFrequency;
        private Text key = new Text();
        @Override
        public void map(IntWritable mapIn,Text mapValIn,Context context) throws IOException,InterruptedException{
            try{
                String line = mapValIn.toString();
                String[] words = line.trim().split("\s");
                List<String> wordList = new ArrayList<String>(Arrays.asList(words));
                String k;
                for(int i=0; i<wordList.size();i++){
                   k = wordList.get(i);
                    //Text key = new Text(k);
                  //  DocAttributes da = new DocAttributes();
                    key.set(k);
                    int sum=1;
                    for(int j=i;j<wordList.size();j++){
                        if(wordList.get(i).matches(wordList.get(j)) && j>i){
                            sum++;
                            docNum = mapIn.get();
                            docFrequency = sum;
                            word = wordList.get(i);
                            termFrequency = sum;
                            wordList.remove(j);
                        }

                        else{
                            docNum = mapIn.get();
                            docFrequency = sum;
                            word = wordList.get(i);
                            termFrequency = sum;
                        }
                    }
                    if(i == words.length-1){
                        docNum = mapIn.get();
                        docFrequency = sum;
                        word = wordList.get(i);
                        termFrequency = sum;
                    }
                    context.write(key, new DocAttributes(docNum,word,docFrequency,termFrequency));
                }
            }
            catch(NullPointerException ne){
                ne.printStackTrace();
            }
        }
    }
    public static class InvertedReduce extends Reducer<Text,DocAttributes,LongWritable,DocAttributes>{
       @Override
        public void reduce(Text key, Iterable<DocAttributes> value,Context context) throws IOException,InterruptedException{
            Iterator<DocAttributes> iterator = value.iterator();
            DocAttributes doc = new DocAttributes();
            List<DocAttributes> list = new ArrayList<DocAttributes>();
            while(iterator.hasNext()){
                list.add(new DocAttributes(iterator.next()));
            }
            Integer docFrequency = 0;
            for(DocAttributes d : list){
                docFrequency += d.getDocFrequency();
                doc.setDocNum(d.getDocNum());
                doc.setWord(d.getWord());
                doc.setDocFrequency(docFrequency);
                doc.setTermFrequency(d.getTermFrequency());
           }
           context.write(new LongWritable(), doc);
       }
    }
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);
        job.setMapperClass(InvertedMap.class);
        // job.setCombinerClass(InvertedCombine.class);
        job.setReducerClass(InvertedReduce.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(DocAttributes.class);
        job.setJarByClass(InvertedIndex.class);
        TextInputFormat.addInputPath(job, new Path(args[0]));
        TextOutputFormat.setOutputPath(job,new Path(args[1]));
        job.setInputFormatClass(DocInput.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        job.submit();
        job.waitForCompletion(true);

    }
}

Answer 1

您正在为每个循环在迭代中使用修改 ArrayList 'wordList'。你能尝试使用迭代器循环在迭代时删除元素（Iterator.remove（））吗？