AvroParquetOutputFormat setSchema方法抛出flink 1.1版本的异常

Question

我的应用程序以拼花格式写入AWS S3a。我的应用程序代码与flink版本1.1-snapshot一起工作，它抛出了与1.1版本而不是快照的异常。

下面的代码用于写入s3

Job job = Job.getInstance();
HadoopOutputFormat hadoopOutputFormat = new HadoopOutputFormat(new AvroParquetOutputFormat(), job);
FileOutputFormat.setOutputPath(job,getOutputPath(runDate));     
AvroParquetOutputFormat.setSchema(job, schema);
AvroParquetOutputFormat.setCompression(job, CompressionCodecName.SNAPPY);
AvroParquetOutputFormat.setEnableDictionary(job, true);
data.output(hadoopOutputFormat);

在设置模式AvroParquetOutputFormat时抛出以下异常。setSchema(工作模式);

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/parquet/ParquetRuntimeException
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.App.writeInS3(App.java:89)
    at com.App.execute(App.java:81)
    at com.App.main(App.java:53)
Caused by: java.lang.ClassNotFoundException: org.apache.parquet.ParquetRuntimeException
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 15 more

Answer 1

如果您正在使用twitter groupId添加到您的hadoop类路径

export HADOOP_CLASSPATH=~/.m2/repository/com/twitter/parquet-avro/1.5.0/parquet-avro-1.5.0.jar::~/.m2/repository/com/twitter/parquet-column/1.5.0/parquet-column-1.5.0.jar:~/.m2/repository/com/twitter/parquet-common/1.5.0/parquet-common-1.5.0.jar:~/.m2/repository/com/twitter/parquet-encoding/1.5.0/parquet-encoding-1.5.0.jar:~/.m2/repository/com/twitter/parquet-generator/1.5.0/parquet-generator-1.5.0.jar:~/.m2/repository/com/twitter/parquet-jackson/1.5.0/parquet-jackson-1.5.0.jar:~/.m2/repository/com/twitter/parquet-format/2.1.0/parquet-format-2.1.0.jar:~/.m2/repository/com/twitter/parquet-hadoop/1.5.0/parquet-hadoop-1.5.0.jar

或者在

中添加其他Parquer依赖项

hadoop_classpath