Write HSSFWorkbook to hdfs

Question

我需要将csv文件解析为xml并将其写入hdfs。我成功地完成了第一部分，但在写作时出现了错误。这是代码。

private static void writeToXml(String inputPath, String outputPath) throws IOException, JSchException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://nn");
FileSystem fileSystem = FileSystem.get(configuration);
Path iPath = new Path(inputPath);
Path oPath = new Path(outputPath);
FSDataInputStream inputStream = fileSystem.open(iPath);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8));
HSSFWorkbook workbook = new HSSFWorkbook();
HSSFSheet sheet = workbook.createSheet("Sheet");
AtomicReference<Integer> row = new AtomicReference<>(0);
try (Stream<String> stream = bufferedReader.lines()) {
stream.forEach(line -> {
Row currentRow = sheet.createRow(row.getAndSet(row.get() + 1));
String[] nextLine = line.split(";");
Stream.iterate(0, i -> i + 1).limit(nextLine.length).forEach(i -> {
currentRow.createCell(i).setCellValue(nextLine[i]);
});
});
FSDataOutputStream outputStream = fileSystem.create(oPath);
ByteArrayOutputStream out = new ByteArrayOutputStream();
workbook.write(out);
outputStream.write(out.toByteArray());
outputStream.flush();
}
}

它因此错误而失败。

org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoSuchMethodError: org.apache.commons.io.IOUtils.byteArray(I)[B
Caused by: java.lang.NoSuchMethodError: org.apache.commons.io.IOUtils.byteArray(I)[B
at org.apache.commons.io.output.AbstractByteArrayOutputStream.needNewBuffer(AbstractByteArrayOutputStream.java:104)
at org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.<init>(UnsynchronizedByteArrayOutputStream.java:51)
at org.apache.poi.poifs.filesystem.POIFSFileSystem.syncWithDataSource(POIFSFileSystem.java:779)
at org.apache.poi.poifs.filesystem.POIFSFileSystem.writeFilesystem(POIFSFileSystem.java:756)
at org.apache.poi.hssf.usermodel.HSSFWorkbook.write(HSSFWorkbook.java:1387)
at path.to.package.Main.writeToXml(Main.java:81)
at path.to.package.Main.main(Main.java:24)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:57)
... 15 more

在这一行。

workbook.write(out);

编辑我尝试写作的另一个剪辑。由于相同错误而失败。

FSDataOutputStream outputStream = fileSystem.create(oPath);
workbook.write(outputStream);
outputStream.flush();

我做错了什么？

Answer 1

最后我找不到我的依赖项有什么问题。我在spark中使用以下依赖项重写了这整件事。

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.crealytics</groupId>
<artifactId>spark-excel_2.11</artifactId>
<version>0.13.7</version>
</dependency>
<dependency>
<groupId>org.apache.xmlbeans</groupId>
<artifactId>xmlbeans</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml-schemas</artifactId>
<version>4.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.14.1</version>
</dependency>
</dependencies>

请注意，xmlbeans是3.1.0版本。由于某种原因，后来的版本对我不起作用。

另外请注意，我只需要spark-excel_2.11就可以在本地进行测试。其他添加是因为它在集群上运行时一直与noClassDefFound一起下降。

代码最终也简化了很多。

spark
.read
.option("delimiter", ";")
.csv("/hdfs/path/to/file.csv")
.repartition(1)
.write
.format("com.crealytics.spark.excel")
.option("header", "false")
.mode("overwrite")
.save("/hdfs/path/to/file.xml")