在DataProc上运行的Java软件中升级Google-Cloud库后，库发生了冲突

Question

我在Google DataProc上运行的Java软件上的Google-Cloud库从0.8.0升级到0.8.0升至0.32.0-alpha版本。

在这里我的依赖性：

<dependencies>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud</artifactId>
        <version>0.32.0-alpha</version>  
    </dependency>
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.2</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.19</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.7.19</version>
     </dependency>
     <dependency>
        <groupId>org.apache.tika</groupId>
        <artifactId>tika-core</artifactId>
        <version>1.12</version>
     </dependency>
     <dependency>
        <groupId>args4j</groupId>
        <artifactId>args4j</artifactId>
        <version>2.33</version>
     </dependency>
     <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
     </dependency>
     <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
     </dependency>
     <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-all</artifactId>
        <version>2.0.2-beta</version>
        <scope>test</scope>
     </dependency>
     <dependency>
         <groupId>javax.mail</groupId>
         <artifactId>mail</artifactId>
         <version>1.4</version>
     </dependency>
     <dependency>
         <groupId>mysql</groupId>
         <artifactId>mysql-connector-java</artifactId>
         <version>5.1.39</version>
     </dependency>
     <dependency>
         <groupId>commons-lang</groupId>
         <artifactId>commons-lang</artifactId>
         <version>2.6</version>
     </dependency>
</dependencies>

这是我在DataProc作业输出中看到的错误

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
    at com.google.api.gax.retrying.BasicRetryingFuture.<init>(BasicRetryingFuture.java:77)
    at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:73)
    at com.google.cloud.RetryHelper.run(RetryHelper.java:73)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
    at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:375)
    at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:366)
    at com.finscience.job.link.LinkAnalyzerProcess.process(LinkAnalyzerProcess.java:210)
    at com.finscience.job.link.LinkAnalyzerJob.main(LinkAnalyzerJob.java:140)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

似乎问题与Google.api.gax库有关。

搜索后，我发现人们解决了该问题，该问题不包括Guava从Maven依赖性出发。因此，我以这种方式修改了我的Google云依赖性：

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud</artifactId>
    <version>0.32.0-alpha</version>
    <exclusions>
       <exclusion>
           <artifactId>com.google</artifactId>
            <groupId>guava</groupId>
       </exclusion>  
    </exclusions>  
 </dependency>

，但不幸的是，这不能解决我的问题。

Google Cloud库的GitHub页面(https://github.com/googlecloodplatform/google-cloud-java(说：

解决版本冲突的最简单方法是使用Google-Cloud的 bom

所以我将以下依赖性添加到我的pom：

<dependency>
   <groupId>com.google.cloud</groupId>
   <artifactId>google-cloud-bom</artifactId>
   <version>0.41.0-alpha</version>
   <type>pom</type>
   <scope>import</scope>
</dependency>

，但即使在这种情况下，问题尚未解决。

有人可以帮我吗？

Answer 1

您最有可能在数据proc集群上与Spark和Hadoop Jars发生冲突。这个答案是如何重新包装罐子来处理Hadoop/Spark冲突。