在使用以前版本的Hadoop时，我通常为整个作业(映射器和reducer)构建junit测试，并通过调用maven或直接从IDE本身运行测试。系统上任何地方都没有安装hadoop。我使用了以下属性来确保使用了本地进程内运行器:
config.set("mapred.job.tracker", "local");
config.set("fs.default.name", "file:///test-fs");
这些测试成功运行。
我升级到新版本的Hadoop(2.4.0)来利用新的API。我希望单位和之前一样。我现在使用的属性是:
config.set("fs.default.name", "file:///test-fs");
config.set("mapreduce.jobtracker.address", "local");
config.set("mapreduce.framework.name", "local");
我在测试中构造配置对象并将其传递给创建Job对象的实际代码。
然而，这不起作用，我得到以下错误:
. io .IOException:无法初始化Cluster。请检查您的配置mapreduce.framework.name和相应的服务器地址。org.apache.hadoop.mapreduce.Cluster.initialize (Cluster.java: 120)在org.apache.hadoop.mapreduce.Cluster。(Cluster.java: 82)在org.apache.hadoop.mapreduce.Cluster。(Cluster.java: 75)在org.apache.hadoop.mapreduce.Job 9.美元运行(Job.java: 1255)在org.apache.hadoop.mapreduce.Job 9.美元运行(Job.java: 1251)在java.security.AccessController。doPrivileged(本地方法)javax.security.auth.Subject.doAs (Subject.java: 415)org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java: 1548)org.apache.hadoop.mapreduce.Job.connect (Job.java: 1250)org.apache.hadoop.mapreduce.Job.submit (Job.java: 1279)org.apache.hadoop.mapreduce.Job.waitForCompletion (Job.java: 1303)com.roblox.games.retention.W1ReturnFromNewPlayersThePreviousWeekCalculatorJob1.calculateW1ReturnJob1 (W1ReturnFromNewPlayersThePreviousWeekCalculatorJob1.java: 68)com.roblox.games.retention.W1ReturnFromNewPlayersThePreviousWeekCalculatorDriver.calculateW1Return (W1ReturnFromNewPlayersThePreviousWeekCalculatorDriver.java: 74)com.roblox.games.retention.W1ReturnFromNewPlayersThePreviousWeekCalculatorDriverTest.testCase (W1ReturnFromNewPlayersThePreviousWeekCalculatorDriverTest.java: 89)在sun.reflect.NativeMethodAccessorImpl。invoke0(本地方法)在sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java: 57)sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java: 43)java.lang.reflect.Method.invoke (Method.java: 606)junit.framework.TestCase.runTest (TestCase.java: 154)junit.framework.TestCase.runBare (TestCase.java: 127)在保护junit.framework.TestResult 1.美元(TestResult.java: 106)junit.framework.TestResult.runProtected (TestResult.java: 124)junit.framework.TestResult.run (TestResult.java: 109)junit.framework.TestCase.run (TestCase.java: 118)junit.framework.TestSuite.runTest (TestSuite.java: 208)junit.framework.TestSuite.run (TestSuite.java: 203)org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run (JUnit3TestReference.java: 131)在org.eclipse.jdt.internal.junit.runner.TestExecution.run (TestExecution.java: 38)org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests (RemoteTestRunner.java: 459)org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests (RemoteTestRunner.java: 675)org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run (RemoteTestRunner.java: 382)org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main (RemoteTestRunner.java: 192)
谁能指出我做错了什么?重申一下，我在运行这些测试的主机上没有任何hadoop安装或二进制文件。我在hadoop 2.7.1中遇到了同样的问题，并通过添加一个依赖项来修复它，该依赖项提供了独立模式的协议实现，类的名称是org.apache.hadoop.mapred.LocalClientProtocolProvider
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>2.7.1</version>
<scope>test</scope>
</dependency>
你不需要设置"mapred.job. "跟踪器"或"fs.default.name"，因为独立运行器是自动从类路径加载的。