我正在尝试对Hadoop2 MapReduce框架进行基准测试。这不是TeraSort。但testmapredsort .
步骤1 创建随机数据:
hadoop jar hadoop/ randomwriter -Dtest.randomwrite.bytes_per_map=100 -Dtest.randomwriter.maps_per_host=10 /data/unsorted-data
step2 对step1中创建的随机数据进行排序:
hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar sort /data/unsorted-data /data/sorted-data
step3 检查MR排序是否有效:
hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar testmapredsort -sortInput /data/unsorted-data -sortOutput /data/sorted-data
我在步骤3中得到以下错误。我想知道如何解决这个错误。
java.lang.Exception: java.io.IOException: Partitions do not match for record# 0 ! - '0' v/s '5'
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:403)
Caused by: java.io.IOException: Partitions do not match for record# 0 ! - '0' v/s '5'
at org.apache.hadoop.mapred.SortValidator$RecordStatsChecker$Map.map(SortValidator.java:266)
at org.apache.hadoop.mapred.SortValidator$RecordStatsChecker$Map.map(SortValidator.java:191)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
14/08/18 11:07:39 INFO mapreduce.Job: Job job_local2061890210_0001 failed with state FAILED due to: NA
14/08/18 11:07:39 INFO mapreduce.Job: Counters: 23
File System Counters
FILE: Number of bytes read=1436271
FILE: Number of bytes written=1645526
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1077294840
HDFS: Number of bytes written=0
HDFS: Number of read operations=13
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=102247
Map output records=102247
Map output bytes=1328251
Map output materialized bytes=26
Input split bytes=102
Combine input records=102247
Combine output records=1
Spilled Records=1
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=22
Total committed heap usage (bytes)=198766592
File Input Format Counters
Bytes Read=1077294840
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
at org.apache.hadoop.mapred.SortValidator$RecordStatsChecker.checkRecords(SortValidator.java:367)
at org.apache.hadoop.mapred.SortValidator.run(SortValidator.java:579)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.mapred.SortValidator.main(SortValidator.java:594)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:115)
at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
编辑:
hadoop fs -ls /data/unsorted-data
-rw-r--r-- 3 david supergroup 0 2014-08-14 12:45 /data/unsorted-data/_SUCCESS
-rw-r--r-- 3 david supergroup 1077294840 2014-08-14 12:45 /data/unsorted-data/part-m-00000
hadoop fs -ls /data/sorted-data
-rw-r--r-- 3 david supergroup 0 2014-08-14 12:55 /data/sorted-data/_SUCCESS
-rw-r--r-- 3 david supergroup 137763270 2014-08-14 12:55 /data/sorted-data/part-m-00000
-rw-r--r-- 3 david supergroup 134220478 2014-08-14 12:55 /data/sorted-data/part-m-00001
-rw-r--r-- 3 david supergroup 134219656 2014-08-14 12:55 /data/sorted-data/part-m-00002
-rw-r--r-- 3 david supergroup 134218029 2014-08-14 12:55 /data/sorted-data/part-m-00003
-rw-r--r-- 3 david supergroup 134219244 2014-08-14 12:55 /data/sorted-data/part-m-00004
-rw-r--r-- 3 david supergroup 134220252 2014-08-14 12:55 /data/sorted-data/part-m-00005
-rw-r--r-- 3 david supergroup 134224231 2014-08-14 12:55 /data/sorted-data/part-m-00006
-rw-r--r-- 3 david supergroup 134210232 2014-08-14 12:55 /data/sorted-data/part-m-00007
除了从test.randomwrite.bytes_per_map和test.randomwriter.maps_per_host到mapreduce.randomwriter.bytespermap和mapreduce.randomwriter.mapsperhost的键的变化导致设置无法通过randomwriter之外，问题的核心是您在/data/sorted-data下列出的文件名所指示的排序数据由map输出组成，而正确排序的输出仅来自reduce输出;实际上，您的sort命令只执行排序的映射部分，而不会在随后的reduce阶段执行合并。因此，您的testmapredsort命令可以正确地报告排序不工作。
检查Sort.java的代码，你可以看到实际上没有防止num_reduces被莫名其妙地设置为0的保护;Hadoop MR的典型行为是将reduce的数量设置为0表示"仅映射"作业，其中映射输出直接到HDFS，而不是作为传递给reduce任务的中间输出。以下是相关内容:
85 int num_reduces = (int) (cluster.getMaxReduceTasks() * 0.9);
86 String sort_reduces = conf.get(REDUCES_PER_HOST);
87 if (sort_reduces != null) {
88 num_reduces = cluster.getTaskTrackers() *
89 Integer.parseInt(sort_reduces);
90 }
现在，在正常设置中，所有使用"默认"设置的逻辑都应该提供非零数量的约简，以便排序工作。我可以通过运行
重现您的问题hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar sort -r 0 /data/unsorted-data /data/sorted-data
使用-r 0强制0减少。在您的情况下，cluster.getMaxReduceTasks()更有可能返回1(如果您的集群损坏，甚至可能返回0)。我不知道这个方法返回1的所有方式;似乎简单地将mapreduce.tasktracker.reduce.tasks.maximum设置为1并不适用于该方法。其他影响任务容量的因素包括内核数量和可用内存量。
假设你的集群每个TaskTracker至少有一个reduce任务，你可以使用-r 1:
hadoop fs -rmr /data/sorted-data
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar sort -r 1 /data/unsorted-data /data/sorted-data