YARN MapReduce没有足够的RAM

Question

我正在尝试在Yarn MapReduce中启动我的应用程序。我已经在 4 个主机（1 个资源管理器和 4 个节点管理器）中设置了 MapReduce。每个主机都有 2 个内核和 4GB 的 RAM。

当我运行我的应用程序时，由于缺少 RAM [1] 而中止。我应该如何设置Yarn MapReduce，以便作业不会耗尽RAM？

[1] 我有错误。

2016-02-09 16:01:41,607 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.6 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:44,612 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:47,616 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:50,621 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:53,626 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 291.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:56,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.1 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:59,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.3 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
 (...)
2016-02-09 16:02:01,173 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/syslog
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stdout
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stderr
2016-02-09 16:02:01,250 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001
2016-02-09 16:02:02,644 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455032058147_0001_01_000001
2016-02-09 16:03:10,146 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:11,147 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:12,148 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:13,149 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Answer 1

您需要配置提供给 YARN 的内存量及其yarn-site.xml中的容器：

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>4096</value>
</property>
<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>1024</value>
</property>

检查此示例纱线站点.xml文件。当你运行MapReduce作业时，你还必须在mapred-site.xml文件中设置类似的属性。本文可能会对您有所帮助 - 链接。