mapreduce中的作业永无止境

Question

我在main方法中设置了一些MapReduce配置，因此

configuration.set("mapreduce.jobtracker.address", "localhost:54311");
configuration.set("mapreduce.framework.name", "yarn");
configuration.set("yarn.resourcemanager.address", "localhost:8032");

现在，当我启动mapreduce任务时，进程会被跟踪（我可以在集群仪表板（监听端口8088的仪表板）中看到它），但进程永远不会结束。它在以下线路保持阻塞：

15/06/30 15:56:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/06/30 15:56:17 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
15/06/30 15:56:18 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/06/30 15:56:18 INFO input.FileInputFormat: Total input paths to process : 1
15/06/30 15:56:18 INFO mapreduce.JobSubmitter: number of splits:1
15/06/30 15:56:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1435241671439_0008
15/06/30 15:56:19 INFO impl.YarnClientImpl: Submitted application application_1435241671439_0008
15/06/30 15:56:19 INFO mapreduce.Job: The url to track the job: http://10.0.0.10:8088/proxy/application_1435241671439_0008/
15/06/30 15:56:19 INFO mapreduce.Job: Running job: job_1435241671439_0008

有人有主意吗？

编辑：在我的yarn nodemanager日志中，我有以下消息

org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_1435241671439_0003_03_000001
2015-06-30 15:44:38,396 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_1435241671439_0002_04_000001

编辑2:

我在yarn manager日志中也有一些较早发生的异常（对于先前的mapreduce调用）：

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [0.0.0.0:8040] java.net.BindException: Address already in use; For more details see: 

解决方案：我杀死了所有守护进程，然后重新启动hadoop！事实上，当我运行jps时，我仍然得到hadoop daemons，尽管我已经阻止了它们。这是HADOOP_PID_DIR 的不匹配

Answer 1

yarn的nodemanage默认端口为8040。错误表明端口已在使用中。停止所有hadoop进程，如果您没有数据，可能会格式化namenode一次，然后再次尝试运行作业。从您的两次编辑来看，问题肯定与节点管理器有关

Answer 2

解决方案：我杀死了所有守护进程，并重新启动hadoop！事实上，当我运行jps时，我仍然得到hadoop守护进程，尽管我已经停止了它们。这与HADOOP_PID_DIR 不匹配有关