从web应用程序(servlet)提交远程Hadoop集群上的Hadoop作业



我已经配置了一个Hadoop集群,其中有两个Linux盒子节点,我有一个servlet应用程序在第三个Linux盒子上的Tomcat上运行。我的目标是从web应用程序提交Hadoop。当我从Eclipse提交作业时,我可以在Hadoop上远程提交作业。在servlet的情况下,我得到以下异常。请建议

'java.net.ConnectException:连接被拒绝:没有进一步的信息;有关更多详细信息,请参阅:http://wiki.apache.org/hadoop/ConnectionRefused在sun.reflect.NativeConstructorAccessorImpl.newInstance0(本机方法)在sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessor Impl.java:39)在sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessor Impl.java:27)位于java.lang.reflect.Constructure.newInstance(Constructor.java:513)网址:org.apache.hadop.net.NetUtils.wrapWithMessage(NetUtils.java:783)网址:org.apache.hadop.net.NetUtils.wrappeException(NetUtils.java:730)网址:org.apache.hadop.ipc.Client.call(Client.java:1414)网址:org.apache.hadop.ipc.Client.call(Client.java:1363)网址:org.apache.hadop.ipc.WriteableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231)位于$Proxy11.getSystemDir(未知源)网址:org.apache.hadop.mapreduce.Cluster$1.run(Cluster.java:141)网址:org.apache.hadop.mapreduce.Cluster$1.run(Cluster.java:139)位于java.security.AccessController.doPrivileged(本机方法)位于javax.security.auth.Subject.doAs(Subject.java:396)网址:org.apache.hadop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)网址:org.apache.hadop.mapreduce.Cluster.getFileSystem(Cluster.java:139)网址:org.apache.hadop.mapreduce.Job.submit(Job.java:1040)网址:org.apache.hadop.mapreduce.Job.waitForCompletion(Job.java:1062)在servlet上。JobDriver.doProcess(JobDriver.java:112)在servlet上。JobDriver.doGet(JobDriver.java:42)位于javax.servlet.httpHttpServlet.service(HttpServlet.java:620)位于javax.servlet.httpHttpServlet.service(HttpServlet.java:727)网址:org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)网址:org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)网址:org.apache.catalina.core.StandardWrapperValv.invoke(StandardWrapperValve.java:220)网址:org.apache.catalina.core.StandardContextValv.invoke(StandardContextValv.java:122)网址:org.apache.catalina.authenticator.AuthenticatorBase.ioke(AuthenticatorBase.java:504)网址:org.apache.catalina.core.StandardHostValv.invoke(StandardHostValve.java:170)网址:org.apache.catalina.vals.ErrorReportValve.ioke(ErrorReportValve.java:103)网址:org.apache.catalina.vals.AccessLogValve.ioke(AccessLogValve.java:950)网址:org.apache.catalina.core.StandardEngineValv.invoke(StandardEngineValv.java:116)网址:org.apache.catalina.connecter.CoyoteAdapter.service(Coyotedapter.java:421)网址:org.apache.coyote.http11.AbstractHttp11Processor.procedure(AbstractHttp11Processer.java:1074)网址:org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)网址:org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)位于java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)位于java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)网址:org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)在java.lang.Thread.run(Thread.java:662)'

从日志来看,进行作业提交的servlet中的用户似乎无权将作业提交到集群:

$Proxy11.getSystemDir(Unknown Source) at 
org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:141) at 
org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:139) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at 
org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:139) at  
org.apache.hadoop.mapreduce.Job.submit(Job.java:1040) at  

通过检查servlet会话,确保只有管理组从servlet提交。

最新更新