org.apache.http.client.utils.URIBuilder在Spark3.2上失败

Question

我正在尝试从Spark 2.4升级到Spark 3.2版本。
import org.apache.http.client.utils.URIBuilder在Spark 3.1上工作，在Spark 3.2上失败，错误为object client is not a member of package org.apache.http。

Answer 1

org.apache.http.client.utils.URIBuilder类来自"org.apache.httpcomponents" % "httpclient"

https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient

你应该加上

libraryDependencies += "org.apache.httpcomponents" % "httpclient" % "4.5.14"

到build.sbt。然后import org.apache.http.client.utils.URIBuilder将编译。

(第5版https://mvnrepository.com/artifact/org.apache.httpcomponents.client5/httpclient5这个类不存在)

---

以下是对原因的解释。

InSpark 2.4.8"org.apache.httpcomponents" % "httpclient"在根pom.xml(<dependencyManagement>)中，没有被覆盖，例如在spark-corepom.xml

<!--  org.apache.httpcomponents/httpclient-->
<commons.httpclient.version>4.5.6</commons.httpclient.version>
...
<dependencyManagement>
<dependencies>
...
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>${commons.httpclient.version}</version>
</dependency>

https://github.com/apache/spark/blob/v2.4.8/pom.xml L499-L503

https://github.com/apache/spark/blob/v2.4.8/core/pom.xml

InSpark 3.2.0这个依赖关系也存在于根pom.xml(<dependencyManagement>)中，但是在spark-corepom.xml(<dependencies>)中被覆盖，因此范围是test

<!--  org.apache.httpcomponents/httpclient-->
<commons.httpclient.version>4.5.13</commons.httpclient.version>
...
<dependencyManagement>
<dependencies>
...
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>${commons.httpclient.version}</version>
</dependency>

<dependencies>
<!-- at least just for tests, coerce SBT to use the updated httpcore/client version -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<scope>test</scope>
</dependency>

https://github.com/apache/spark/blob/v3.2.0/pom.xml L622-L626

https://github.com/apache/spark/blob/v3.2.0/core/pom.xml L370-L375

所以，如果你需要这个依赖项，不仅在test范围内，你应该手动添加它。

Maven中dependencyManagement和dependencies的区别