我是Hadoop的新手。我的笔记本电脑是32GB，酷睿i5,4核处理器。我已经通过虚拟机创建了多节点（3 个数据节点）apache hadoop 集群 2.7.4。我分配了 8GB，每个数据节点 2 个核心 CPU，资源管理器虚拟机。当我在namenode上运行mapreducehadoop示例作业时，几乎每次我的作业都由于Map任务或reduce任务失败而失败。
我在日志中没有看到任何特定错误，但请注意所有地图和 减少容器尝试在同一数据节点上查找容器的任务，如果 它失败了几次，然后应用程序主节点选择另一个节点 对于可用的容器。
有没有办法像轮询方式一样在数据节点上分配容器？
任何帮助都是可观的。
输出-
hduser@NameNode:/opt/hadoop/etc/hadoop$ hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar pi 2 4
Number of Maps = 2
Samples per Map = 4
Wrote input for Map #0
Wrote input for Map #1
Starting Job
.....
17/11/02 12:53:33 INFO mapreduce.Job: Running job: job_1509607315241_0001
17/11/02 12:53:40 INFO mapreduce.Job: Job job_1509607315241_0001 running in uber mode : false
17/11/02 12:53:40 INFO mapreduce.Job: map 0% reduce 0%
17/11/02 12:53:55 INFO mapreduce.Job: Task Id : attempt_1509607315241_0001_m_000001_0, Status : FAILED
17/11/02 12:53:55 INFO mapreduce.Job: Task Id : attempt_1509607315241_0001_m_000000_0, Status : FAILED
17/11/02 12:54:01 INFO mapreduce.Job: map 50% reduce 0%
17/11/02 12:54:09 INFO mapreduce.Job: Task Id : attempt_1509607315241_0001_m_000001_1, Status : FAILED
17/11/02 12:54:14 INFO mapreduce.Job: Task Id : attempt_1509607315241_0001_r_000000_0, Status : FAILED
17/11/02 12:54:24 INFO mapreduce.Job: Task Id : attempt_1509607315241_0001_m_000001_2, Status : FAILED
17/11/02 12:54:30 INFO mapreduce.Job: Task Id : attempt_1509607315241_0001_r_000000_1, Status : FAILED
17/11/02 12:54:40 INFO mapreduce.Job: map 100% reduce 100%
17/11/02 12:54:44 INFO mapreduce.Job: Job job_1509607315241_0001 failed with state FAILED due to: Task failed task_1509607315241_0001_m_000001
Yarn-site.xnl
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>192.168.10.109</value>
<description> The hostname of the machine the resource manager runs on. </description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>A list of auxiliary services run by the node manager. A service is implemented by the class defined by the property yarn.nodemanager.auxservices.servicename.class. By default, no auxiliary services are specified. </description>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
<description> </description>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>7096</value>
<description>The amount of physical memory (in MB) that may be allocated to containers being run by the node manager </description>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>6196</value>
<description>RM can only allocate memory to containers in increments of "yarn.scheduler.minimum-allocation-mb" and not exceed "yarn.scheduler.maximum-allocation-mb" and It should not be more then total allocated memory of the Node. </description>
</property>
<property>
<name>yarn.nodemanager.delete.debug-delay-sec</name>
<value>6000</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
<description>RM can only allocate memory to containers in increments of "yarn.scheduler.minimum-allocation-mb" and not exceed "yarn.scheduler.maximum-allocation-mb" and It should not be more then total allocated memory of the Node. </description>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx2048m</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
<description>The number of CPU cores that may be allocated to containers being run by the node manager.</description>
</property>
<property>
<name>yarn.resourcemanager.bind-host</name>
<value>192.168.10.109</value>
<description> The address the resource manager’s RPC and HTTP servers will bind to.</description>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>192.168.10.109:8032</value>
<description>The hostname and port that the resource manager’s RPCserver runs on. </description>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>192.168.10.109:8033</value>
<description>The resource manager’s admin RPC server address and port. This is used by the admin client (invoked with yarn rmadmin, typically run outside the cluster) to communicate with the resource manager. </description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>192.168.10.109:8030</value>
<description>The resource manager scheduler’s RPC server address and port. This is used by (in-cluster) application masters to communicate with the resource manager.</description>
</property>
<property>
<name>yarn.resourcemanager.resourcetracker.address</name>
<value>192.168.10.109:8031</value>
<description>The resource manager resource tracker’s RPC server address and port. This is used by (incluster) node managers to communicate with the resource manager. </description>
</property>
<property>
<name>yarn.nodemanager.hostname</name>
<value>0.0.0.0</value>
<description>The hostname of the machine the node manager runs on. </description>
</property>
<property>
<name>yarn.nodemanager.bind-host</name>
<value>0.0.0.0</value>
<description>The address the node manager’s RPC and HTTP servers will bind to. </description>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/opt/hadoop/hdfs/yarn</value>
<description>A list of directories where nodemanagers allow containers to store intermediate data. The data is cleared out when the application ends.</description>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:8050</value>
<description>The node manager’s RPC server address and port. This is used by (in-cluster) application masters to communicate with node managers.</description>
</property>
<property>
<name>yarn.nodemanager.localizer.address</name>
<value>0.0.0.0:8040</value>
<description>The node manager localizer’s RPC server address and port. </description>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>192.168.10.109:8088</value>
<description> The resource manager’s HTTP server address and port.</description>
</property>
<property>
<name>yarn.nodemanager.webapp.address</name>
<value>0.0.0.0:8042</value>
<description>The node manager’s HTTP server address and port. </description>
</property>
<property>
<name>yarn.web-proxy.address</name>
<value>192.168.10.109:9046</value>
<description>The web app proxy server’s HTTP server address and port. If not set (the default), then the web app proxy server will run in the resource manager process. MapReduce ApplicationMaster REST APIs are accessed using a proxy server, that is, Web Application Proxy server. Proxy server is an optional service in YARN. An administrator can configure the
service to run on a particular host or on the ResourceManager itself (stand-alone mode). If the proxy server is not configured, then it runs as a part of the ResourceManager service. By default, REST calls could be made to the web address port of
ResourceManager 8088. </description>
</property>
</configuration>
地图网站.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Default framework to run.</description>
</property>
<!-- <property>
<name>mapreduce.jobtracker.address</name>
<value>localhost:54311</value>
<description>MapReduce job tracker runs at this host and port.</description>
</property> -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>192.168.10.109:19888</value>
<description>The MapReduce job history server’s addressand port.</description>
</property>
<property>
<name>mapreduce.shuffle.port</name>
<value>13562</value>
<description>The shuffle handler’s HTTP port number.This is used for serving map outputs, and is not a user-accessible web UI.</description>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>192.168.10.109:10020</value>
<description>The job history server’s RPC server address and port. This is used by the client (typically outside the cluster) to query job history.</description>
</property>
<property>
<name>mapreduce.jobhistory.bind-host</name>
<value>192.168.10.109</value>
<description>Setting all of these values to 0.0.0.0 as in the example above will cause the MapReduce daemons to listen on all addresses and interfaces of the hosts in the cluster.</description>
</property>
<property>
<name>mapreduce.job.userhistorylocation</name>
<value>/opt/hadoop/hdfs/mrjobhistory</value>
<description>User can specify a location to store the history files of a particular job. If nothing is specified, the logs are stored in output directory. The files are stored in "_logs/history/" in the directory. User can stop logging by giving the value "none".</description>
</property>
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/opt/hadoop/hdfs/mrjobhistory/tmp</value>
<description>Directory where history files are written by MapReduce jobs.</description>
</property>
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/opt/hadoop/hdfs/mrjobhistory/done</value>
<description>Directory where history files are managed by the MR JobHistory Server.</description>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>
<property>
<name>mapreduce.map.cpu.vcores</name>
<value>1</value>
<description> The number of virtual cores to request from the scheduler for each map task.</description>
</property>
<property>
<name>mapreduce.reduce.cpu.vcores</name>
<value>1</value>
<description> The number of virtual cores to request from the scheduler for each reduce task.</description>
</property>
<property>
<name>mapreduce.task.timeout</name>
<value>1800000</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1555m</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2048m</value>
</property>
<property>
<name>mapreduce.job.running.map.limit</name>
<value>2</value>
<description> The maximum number of simultaneous map tasks per job. There is no limit if this value is 0 or negative.</description>
</property>
<property>
<name>mapreduce.job.running.reduce.limit</name>
<value>1</value>
<description> The maximum number of simultaneous reduce tasks per job. There is no limit if this value is 0 or negative.</description>
</property>
<property>
<name>mapreduce.reduce.shuffle.connect.timeout</name>
<value>1800000</value>
<description>Expert: The maximum amount of time (in milli seconds) reduce task spends in trying to connect to a tasktracker for getting map output.</description>
</property>
<property>
<name>mapreduce.reduce.shuffle.read.timeout</name>
<value>1800000</value>
<description>Expert: The maximum amount of time (in milli seconds) reduce task waits for map output data to be available for reading after obtaining connection.</description>
</property>
<!--
<property>
<name>mapreduce.job.reducer.preempt.delay.sec</name>
<value>300</value>
<description> The threshold (in seconds) after which an unsatisfied mapper request triggers reducer preemption when there is no anticipated headroom. If set to 0 or a negative value, the reducer is preempted as soon as lack of headroom is detected. Default is 0.</description>
</property>
<property>
<name>mapreduce.job.reducer.unconditional-preempt.delay.sec</name>
<value>400</value>
<description> The threshold (in seconds) after which an unsatisfied mapper request triggers a forced reducer preemption irrespective of the anticipated headroom. By default, it is set to 5 mins. Setting it to 0 leads to immediate reducer preemption. Setting to -1 disables this preemption altogether.</description>
</property>
-->
</configuration>
问题出在数据节点中的/etc/hosts 文件上。我们必须注释指向其环回地址的主机名所在的行。我已经用日志中的行追踪了这个错误-
INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at DN1/127.0.1.1:54483
早些时候
127.0.1.1 DN1
192.168.10.104 dn1
后
# 127.0.1.1 DN1
192.168.10.104 DN1
我建议在mapred-site.xml中添加以下属性：
<property>
<name>mapreduce.map.maxattempts</name>
<value>20</value>
</property>
<property>
<name>mapreduce.reduce.maxattempts</name>
<value>20</value>
</property>