我正试图在一个没有管理权限的集群上使用MRJob运行python脚本,我在下面粘贴了错误。我认为正在发生的情况是,作业正在尝试将中间文件写入默认的/tmp。。。。dir,由于这是一个受保护的目录,我没有写入权限,所以作业收到一个错误并退出。我想知道如何将这个tmp输出目录位置更改为本地文件系统中的某个位置。例如:/home/myusername/some_path_in_my_local_filesystem_on_the_cluster
,基本上我想知道我必须传递哪些附加参数才能将中间输出位置从/tmp/更改为。。。去当地某个我有写作许可的地方。
我调用我的脚本为:
python myscript.py input.txt -r hadoop > output.txt
错误:
no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
creating tmp directory /tmp/13435.1.all.q/mr_word_freq_count.myusername.20131215.004905.274232
writing wrapper script to /tmp/13435.1.all.q/mr_word_freq_count.myusername.20131215.004905.274232/setup-wrapper.sh
STDERR: mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: user=myusername, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
Traceback (most recent call last):
File "/home/myusername/privatemodules/python/examples/mr_word_freq_count.py", line 37, in <module>
MRWordFreqCount.run()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/job.py", line 500, in run
mr_job.execute()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/job.py", line 518, in execute
super(MRJob, self).execute()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/launch.py", line 146, in execute
self.run_job()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/launch.py", line 207, in run_job
runner.run()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/runner.py", line 458, in run
self._run()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/hadoop.py", line 236, in _run
self._upload_local_files_to_hdfs()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/hadoop.py", line 263, in _upload_local_files_to_hdfs
self._mkdir_on_hdfs(self._upload_mgr.prefix)
您是将mrjob作为"本地"作业运行,还是尝试在Hadoop集群上运行它?
如果您实际尝试在Hadoop上使用它,您可以使用--base-tmp-dir标志:来控制"scratch"HDFS位置(mrjob将在其中存储中间文件)
python mr.py -r hadoop -o hdfs:///user/you/output_dir --base-tmp-dir hdfs:///user/you/tmp hdfs:///user/you/data.txt