Hadoop YARN -如何限制请求内存

Question

尝试从hadoop-mapreduce-examples-2.2.0.jar运行PI示例，我得到以下例外:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=1536, maxMemory=512

不确定1536来自哪里，但512是我在mapred-site.xml中设置的子任务的最大堆大小:

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>512</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx410m</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>512</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx410m</value>
</property>

确定map/reduce任务大小的正确方法是什么?

Answer 1

512为yarn-site.xml中yarn.scheduler.maximum-allocation-mb参数的默认值，1536为mapred-site.xml中yarn.app.mapreduce.am.resource.mb参数的默认值。

确保allocation-mb> app.mapreduce.am.resource.mb，这样就可以了