"Counters from Step 1: No Counters found"使用Hadoop和mrjob时

Question

我有一个python文件来计数bigrams使用mrjob在Hadoop(版本2.6.0)上，但我没有得到我所希望的输出，我在我的终端中破译我出错的地方有困难。

我的代码:

regex_for_words = re.compile(r"b[w']+b")
class BiCo(MRJob):
  OUTPUT_PROTOCOL = mrjob.protocol.RawProtocol
  def mapper(self, _, line):
    words = regex_for_words.findall(line)
    wordsinline = list()
    for word in words:
        wordsinline.append(word.lower()) 
    wordscounter = 0
    totalwords = len(wordsinline)
    for word in wordsinline:
        if wordscounter < (totalwords - 1):
            nextword_pos = wordscounter+1
            nextword = wordsinline[nextword_pos]
            bigram = word, nextword
            wordscounter +=1
            yield (bigram, 1)
  def combiner(self, bigram, counts):
    yield (bigram, sum(counts))
  def reducer(self, bigram, counts):
    yield (bigram, str(sum(counts)))
if __name__ == '__main__':
  BiCo.run()

我在本地机器上编写了mapper函数中的代码(基本上，从"yield"行开始的所有代码)，以确保我的代码按预期抓取双元数据，所以我认为它应该工作得很好....但是，当然，有些地方出了问题。

当我在Hadoop服务器上运行代码时，我得到以下输出(如果这是不必要的，请道歉-屏幕输出大量信息，我还不确定什么将有助于在问题区域进行研究):

HADOOP: 2015-10-25 17:00:46,992 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1355)) - Running job: job_1438612881113_6410
HADOOP: 2015-10-25 17:00:52,110 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1376)) - Job job_1438612881113_6410 running in uber mode : false
HADOOP: 2015-10-25 17:00:52,111 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1383)) -  map 0% reduce 0%
HADOOP: 2015-10-25 17:00:58,171 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1383)) -  map 33% reduce 0%
HADOOP: 2015-10-25 17:01:00,184 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1383)) -  map 100% reduce 0%
HADOOP: 2015-10-25 17:01:07,222 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1383)) -  map 100% reduce 100%
HADOOP: 2015-10-25 17:01:08,239 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1394)) - Job job_1438612881113_6410 completed successfully
HADOOP: 2015-10-25 17:01:08,321 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1401)) - Counters: 51
HADOOP:         File System Counters
HADOOP:                 FILE: Number of bytes read=2007840
HADOOP:                 FILE: Number of bytes written=4485245
HADOOP:                 FILE: Number of read operations=0
HADOOP:                 FILE: Number of large read operations=0
HADOOP:                 FILE: Number of write operations=0
HADOOP:                 HDFS: Number of bytes read=1013129
HADOOP:                 HDFS: Number of bytes written=0
HADOOP:                 HDFS: Number of read operations=12
HADOOP:                 HDFS: Number of large read operations=0
HADOOP:                 HDFS: Number of write operations=2
HADOOP:         Job Counters
HADOOP:                 Killed map tasks=1
HADOOP:                 Launched map tasks=4
HADOOP:                 Launched reduce tasks=1
HADOOP:                 Rack-local map tasks=4
HADOOP:                 Total time spent by all maps in occupied slots (ms)=33282
HADOOP:                 Total time spent by all reduces in occupied slots (ms)=12358
HADOOP:                 Total time spent by all map tasks (ms)=16641
HADOOP:                 Total time spent by all reduce tasks (ms)=6179
HADOOP:                 Total vcore-seconds taken by all map tasks=16641
HADOOP:                 Total vcore-seconds taken by all reduce tasks=6179
HADOOP:                 Total megabyte-seconds taken by all map tasks=51121152
HADOOP:                 Total megabyte-seconds taken by all reduce tasks=18981888
HADOOP:         Map-Reduce Framework
HADOOP:                 Map input records=28214
HADOOP:                 Map output records=133627
HADOOP:                 Map output bytes=2613219
HADOOP:                 Map output materialized bytes=2007852
HADOOP:                 Input split bytes=304
HADOOP:                 Combine input records=133627
HADOOP:                 Combine output records=90382
HADOOP:                 Reduce input groups=79518
HADOOP:                 Reduce shuffle bytes=2007852
HADOOP:                 Reduce input records=90382
HADOOP:                 Reduce output records=0
HADOOP:                 Spilled Records=180764
HADOOP:                 Shuffled Maps =3
HADOOP:                 Failed Shuffles=0
HADOOP:                 Merged Map outputs=3
HADOOP:                 GC time elapsed (ms)=93
HADOOP:                 CPU time spent (ms)=7940
HADOOP:                 Physical memory (bytes) snapshot=1343377408
HADOOP:                 Virtual memory (bytes) snapshot=14458105856
HADOOP:                 Total committed heap usage (bytes)=4045406208
HADOOP:         Shuffle Errors
HADOOP:                 BAD_ID=0
HADOOP:                 CONNECTION=0
HADOOP:                 IO_ERROR=0
HADOOP:                 WRONG_LENGTH=0
HADOOP:                 WRONG_MAP=0
HADOOP:                 WRONG_REDUCE=0
HADOOP:         Unencodable output
HADOOP:                 TypeError=79518
HADOOP:         File Input Format Counters
HADOOP:                 Bytes Read=1012825
HADOOP:         File Output Format Counters
HADOOP:                 Bytes Written=0
HADOOP: 2015-10-25 17:01:08,321 INFO  [main] streaming.StreamJob (StreamJob.java:submitAndMonitorJob(1022)) - Output directory: hdfs:///user/andersaa/si601f15lab5_output
Counters from step 1:
  (no counters found)

我很困惑为什么从第1步没有找到计数器(我假设是代码的映射器部分，这可能是一个错误的假设)。如果我正确地阅读Hadoop的任何输出，看起来它至少进入了reduce阶段(因为有reduce Input组)，并且没有发现任何shuffle错误。我认为可能有一些答案是什么出了问题在"不可编码的输出:TypeError=79518"，但没有多少谷歌搜索，我已经做了帮助磨练这是什么错误。

非常感谢任何帮助或见解。

Answer 1

一个问题是在映射器的双字符编码中。上面的编码方式，bigram是python类型"tuple":

>>> word = 'the'
>>> word2 = 'boy'
>>> bigram = word, word2
>>> type(bigram)
<type 'tuple'>

通常，普通字符串用作键。因此，将bigram创建为字符串。一种方法是:

bigram = '-'.join((word, nextword))

当我在你的程序中做了这个改变，然后我看到这样的输出:

automatic-translation   1
automatic-vs    1
automatically-focus 1
automatically-learn 1
automatically-learning  1
automatically-translate 1
available-including 1
available-without   1

另一个提示:尝试在命令行上使用-q来静音所有hadoop中间噪声。有时它只是阻碍。

HTH .

Answer 2

缓存错误。我在《Hortonworks sandbox》中发现了这一点。简单的解决方案是从沙箱中注销并再次ssh .