>2018-03-08 16:36:16,775 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: 正在下载公共 rsrc:{ hdfs://mycluster/user/abc_user/udf/pig_udf-1.5.7_handle_input_error.jar, 1516336589685, 文件, null } 2018-03-08 16:36:16,775 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: 无法下载资源 { { hdfs://mycluster/user/oozie/share/lib/lib_20171215093741/pig/libgplcompression.so.0.0.0, 1513307849411, FILE, null },待定,[(container_1519371600813_0002_02_000001)],8140205165392614,下载}java.lang.IllegalArgumentException: java.net.UnknownHostException: myclusterat org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:728) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:671) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:155) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2852) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2834) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:249) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecumytor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) 原因:java.net.UnknownHost异常:mycluster
Yarn-nodemanager 服务和数据节点服务在同一台机器上
Yarn-resource-manager 服务和 NameNode 位于同一台机器上
当运行一个简单的pig脚本加载数据并打印.我遇到了上面的错误.
在添加备用 Namnode 之前,一切正常。
如何配置纱线来理解我的NameNode集群
谢谢
再次检查 hdfs-site.xml 在 2 个 Yarn 节点管理器所在的数据节点上,我看到 hdfs-site 文件在与名称节点上的 hdfs 站点进行比较时缺少此行
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
它现在工作