Hadoop - MultipleInputs

Question

我正在尝试使用Hadoop中的MultipleInputs。我的所有映射器都将是FixedLengthInputFormat。

MultipleInputs.addInputPath(job, 
                    new Path(rootDir),       
                    FixedLengthInputFormat.class, 
                    OneToManyMapper.class);

问题是，每个映射器都有固定的记录宽度和不同的大小。

config.setInt（FixedLengthInputFormat.FIXED_RECORD_LENGTH，？）；

是否使用MultipleInputs为每个映射器传递FIXED_RECORD_LENGTH？

谢谢！

Answer 1

以下是解决方案：

public class CustomFixedLengthInputFormat extends FixedLengthInputFormat{
    @Override
    public RecordReader<LongWritable, BytesWritable> createRecordReader(
            InputSplit split, TaskAttemptContext context) throws IOException,
            InterruptedException {
        //here i can control de recordLength size!
        int recordLength = ??;// getRecordLength(context.getConfiguration());
        if (recordLength <= 0) {
            throw new IOException(
                    "Fixed record length "
                            + recordLength
                            + " is invalid.  It should be set to a value greater than zero");
        }
        System.out.println("Record Length: " + recordLength);
        return new FixedLengthRecordReader(recordLength);
    }
}