无法在EC2中使用S3进行第二次运行MapReduce进行存储



我在 Amazon EC2 实例中设置了单节点 hadoop。遵循这个和这个,我可以第一次运行示例程序。但是要让它第二次运行它,我必须在停止发出 stop-all.sh 后删除 S3 和本地 tmp 目录中的所有目录和文件。我只运行mapred(任务跟踪器和工作跟踪器)。尝试第二次重新运行该示例时,我收到错误消息。

        hduser@ip-10-252-196-143:~$ hadoop jar ./hadoop/hadoop-examples-1.2.1.jar  wordcount input output2
13/09/20 09:43:06 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.IOException: The ownership on the staging directory s3://vikesh-hadoop-bucket/home/hduser/tmp/mapred/staging/hduser/.staging is not as expected. It is owned by . The directory must be owned by the submitter hduser or by hduser
java.io.IOException: The ownership on the staging directory s3://vikesh-hadoop-bucket/home/hduser/tmp/mapred/staging/hduser/.staging is not as expected. It is owned by . The directory must be owned by the submitter hduser or by hduser
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

似乎这是相关的,但我相信正在使用Cloudera的发行版。

为什么会发生这种情况,我能做些什么来解决这个问题?我在 64 位 VM 上使用 hadoop-1.2.1 和 OpenJDK 7。

谢谢

我们遇到了同样的问题。我们只需删除每个 MapReduce 之间的 s3://{bucket-name}/home/{user}/tmp/mapred/staging/{user}/.staging...但是如果有人有更好的解决方案...

相关内容

  • 没有找到相关文章

最新更新