Mapreduce矩阵乘法与hadoop

我试图在以下链接上运行提到的矩阵乘法示例(带有源代码):

http://www.norstad.org/matrix-multiply/index.html

我在伪分布式模式下安装hadoop，我使用这个教程配置它:

http://hadoop-tutorial.blogspot.com/2010/11/running-hadoop-in-pseudo-distributed.html?showComment=1321528406255 c3661776111033973764

当我运行jar文件时，我得到以下错误:

Identity test
11/11/30 10:37:34 INFO input.FileInputFormat: Total input paths to process : 2
11/11/30 10:37:34 INFO mapred.JobClient: Running job: job_201111291041_0010
11/11/30 10:37:35 INFO mapred.JobClient:  map 0% reduce 0%
11/11/30 10:37:44 INFO mapred.JobClient:  map 100% reduce 0%
11/11/30 10:37:56 INFO mapred.JobClient:  map 100% reduce 100%
11/11/30 10:37:58 INFO mapred.JobClient: Job complete: job_201111291041_0010
11/11/30 10:37:58 INFO mapred.JobClient: Counters: 17
11/11/30 10:37:58 INFO mapred.JobClient:   Job Counters
11/11/30 10:37:58 INFO mapred.JobClient:     Launched reduce tasks=1
11/11/30 10:37:58 INFO mapred.JobClient:     Launched map tasks=2
11/11/30 10:37:58 INFO mapred.JobClient:     Data-local map tasks=2
11/11/30 10:37:58 INFO mapred.JobClient:   FileSystemCounters
11/11/30 10:37:58 INFO mapred.JobClient:     FILE_BYTES_READ=114
11/11/30 10:37:58 INFO mapred.JobClient:     HDFS_BYTES_READ=248
11/11/30 10:37:58 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=298
11/11/30 10:37:58 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=124
11/11/30 10:37:58 INFO mapred.JobClient:   Map-Reduce Framework
11/11/30 10:37:58 INFO mapred.JobClient:     Reduce input groups=2
11/11/30 10:37:58 INFO mapred.JobClient:     Combine output records=0
11/11/30 10:37:58 INFO mapred.JobClient:     Map input records=4
11/11/30 10:37:58 INFO mapred.JobClient:     Reduce shuffle bytes=60
11/11/30 10:37:58 INFO mapred.JobClient:     Reduce output records=2
11/11/30 10:37:58 INFO mapred.JobClient:     Spilled Records=8
11/11/30 10:37:58 INFO mapred.JobClient:     Map output bytes=100
11/11/30 10:37:58 INFO mapred.JobClient:     Combine input records=0
11/11/30 10:37:58 INFO mapred.JobClient:     Map output records=4
11/11/30 10:37:58 INFO mapred.JobClient:     Reduce input records=4
11/11/30 10:37:58 INFO input.FileInputFormat: Total input paths to process : 1
11/11/30 10:37:59 INFO mapred.JobClient: Running job: job_201111291041_0011
11/11/30 10:38:00 INFO mapred.JobClient:  map 0% reduce 0%
11/11/30 10:38:09 INFO mapred.JobClient:  map 100% reduce 0%
11/11/30 10:38:21 INFO mapred.JobClient:  map 100% reduce 100%
11/11/30 10:38:23 INFO mapred.JobClient: Job complete: job_201111291041_0011
11/11/30 10:38:23 INFO mapred.JobClient: Counters: 17
11/11/30 10:38:23 INFO mapred.JobClient:   Job Counters
11/11/30 10:38:23 INFO mapred.JobClient:     Launched reduce tasks=1
11/11/30 10:38:23 INFO mapred.JobClient:     Launched map tasks=1
11/11/30 10:38:23 INFO mapred.JobClient:     Data-local map tasks=1
11/11/30 10:38:23 INFO mapred.JobClient:   FileSystemCounters
11/11/30 10:38:23 INFO mapred.JobClient:     FILE_BYTES_READ=34
11/11/30 10:38:23 INFO mapred.JobClient:     HDFS_BYTES_READ=124
11/11/30 10:38:23 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=100
11/11/30 10:38:23 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=124
11/11/30 10:38:23 INFO mapred.JobClient:   Map-Reduce Framework
11/11/30 10:38:23 INFO mapred.JobClient:     Reduce input groups=2
11/11/30 10:38:23 INFO mapred.JobClient:     Combine output records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Map input records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Reduce shuffle bytes=0
11/11/30 10:38:23 INFO mapred.JobClient:     Reduce output records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Spilled Records=4
11/11/30 10:38:23 INFO mapred.JobClient:     Map output bytes=24
11/11/30 10:38:23 INFO mapred.JobClient:     Combine input records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Map output records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Reduce input records=2
Exception in thread "main" java.io.IOException: Cannot open filename /tmp/Matrix Multiply/out/_logs
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.ja va:1497)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java :1488)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:376)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSyst em.java:178)
        at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1 437)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:142 4)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:141 7)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:141 2)
        at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:62)
        at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:84)
        at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:108)
        at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:144)
        at TestMatrixMultiply.testIdentity(TestMatrixMultiply.java:156)
        at TestMatrixMultiply.main(TestMatrixMultiply.java:258)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

谁能告诉我，我做错了什么?由于

它尝试读取作业输出。当您将其提交到集群时，它将添加这个_log目录。因为目录不是序列文件，所以它们不能被读取。

您必须更改读取此内容的代码。

我写了一个相等的脚本:

FileStatus[] stati = fs.listStatus(output);
for (FileStatus status : stati) {
    if (!status.isDir()) {
        Path path = status.getPath();
        // HERE IS THE READ CODE FROM YOUR EXAMPLE
    }
}

http://code.google.com/p/hama-shortest-paths/source/browse/trunk/hama-gsoc/src/de/jungblut/clustering/mapreduce/KMeansClusteringJob.java 127

这可能是一个原始的建议，但是，您可能需要使用/tmp/矩阵//_logs繁殖。目录名中的空格可能不会自动处理，我假设您在Linux上工作。

testmatrixmultiple .java有两个问题:

正如Thomas Jungblut所说，在readMatrix()方法中应该排除_logs。我把代码改成这样:

if (fs.isFile(path)) {
        fillMatrix(result, path);
    } else {
        FileStatus[] fileStatusArray = fs.listStatus(path);
        for (FileStatus fileStatus : fileStatusArray) {
            if ( !fileStatus.isDir() )  // this line is added by me
                fillMatrix(result, fileStatus.getPath());
        }
    }

在main()方法的末尾，fs.delete应该被注释，否则每次mapreduce任务完成后，输出目录会被立即删除。
```
finally {
        //fs.delete(new Path(DATA_DIR_PATH), true);
    }
```

相关内容

最新更新

热门标签：