如何在 MRUnit 中使用 avro 参数测试减速器



我正在尝试测试减速器波纹管

Reducer<CompositeKeyWritable, AvroValue<Class1>, AvroKey<Class2>, NullWritable>

为此,我在测试中使用下一个代码:

CustomReducer reducer = new CustomReducer();
    reduceDriver = new ReduceDriver<CompositeKeyWritable, AvroValue<Class1>, AvroKey<Class2>, NullWritable>();
    Job job = new Job();
    reduceDriver = ReduceDriver.newReduceDriver(reducer).withConfiguration(job.getConfiguration());
    reduceDriver.setReducer(reducer);
    job.setInputFormatClass(AvroKeyInputFormat.class);
    job.setMapOutputKeyClass(CompositeKeyWritable.class);
    AvroJob.setMapOutputValueSchema(job, union);
    AvroJob.setOutputKeySchema(job, Class2.SCHEMA$);
    job.setOutputFormatClass(AvroKeyOutputFormat.class);
    job.setOutputValueClass(NullWritable.class);

结果我得到了下一个错误:

java.lang.RuntimeException: java.io.EOFException
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:86)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:110)
at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:675)
at org.apache.hadoop.mrunit.ReduceDriverBase.addInput(ReduceDriverBase.java:167)
at org.apache.hadoop.mrunit.ReduceDriverBase.addInput(ReduceDriverBase.java:181)
at com.wellcare.caregap.hadoop.reducers.CareGapReducerTest.testReducer(CareGapReducerTest.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

作业配置有什么问题?如果有人帮助我解决这些例外,我将不胜感激。

调试你的方法 readFields 并在 Class1 中写入。当您使用reduceDriver.withInput(yourkey, yourvalue);添加输入时,它似乎在到达其中一个 DataInput 或 DataOutput 流的末尾后尝试读取或写入字段

相关内容

  • 没有找到相关文章

最新更新