number of vcore in yarn

Question

我是一个纱线新手。我在电脑上安装了单节点Hadoop Yarn，一切都是默认的。我打开ResourceManager的网页，它说我的集群是8GB RAM和8个内核。但实际上，我的计算机有3GB的ram和2个内核(我在/proc/meminfo和/proc/cpuinfo中看到信息)。我想知道为什么纱线显示太多的ram和vcore?大家能给我解释一下吗?提前感谢!

Answer 1

Yarn通过以下两个参数从Yarn -site.xml配置文件中获取可用内存和内核的信息:

<!-- Default 8GB -->
<property>
  <description>Max available memory on each data node.</description>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>8192</value>
</property>
<!-- Default 8. -->
<property>
  <description>Max available cores data node.</description>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>8</value>
</property>

所以它选择8GB和8核的原因是因为如果你不指定其他任何东西，这些都是默认值。这些设置对于像你这样的小机器来说可能太高了。在小型机器上运行时，您可能需要调整一些其他与内存相关的设置。

在yarn-site.xml:

<!-- Default 1024 -->
<property>
  <description>Minimum allocation unit.</description>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>256</value>
</property>
<!-- Biggest memory allocation a container can request. Set to available memory -->
<property>
  <description>Max allocation unit.</description>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>yarnavailablememory</value>
</property>
<property>
  <description>Minimum increment setting - set to same as min-allocation</description>
  <name>yarn.scheduler.increment-allocation-mb</name>
  <value>256</value>
</property>

在mapred-site.xml:

<!-- small cluster memory settings -->
<!-- Default 1024. Recommend setting to 4096. Should not be higher than YARN max allocation -->
<property>
  <name>mapreduce.map.memory.mb</name>
  <value>256</value>
</property>
<!-- Default 1024. Recommend setting to 4096. Should not be higher than YARN max allocation -->
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>256</value>
</property>
<!-- Default 1536. Recommend 1024 -->
<property>
  <description>Application master allocation</description>
  <name>yarn.app.mapreduce.am.resource.mb</name>
  <value>256</value>
</property>
<!-- Recommend heapsizes to be 75% of mapreduce.map/reduce.memory.mb -->
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx204m</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx204m</value>
</property> 
<property>
  <description>Application Master JVM opts</description>
  <name>yarn.app.mapreduce.am.command-opts</name>
  <value>-Xmx204m</value>
</property>