使用Java连接EMR HBase

Question

我试图从运行在EMR 5.35 (Hadoop 2.10, Spark 2.4.8, HBase 1.4.13)上的Spark程序内部连接到HBase当不尝试连接HBase时，我的Spark程序运行得很好。

但是，当我添加HBase代码时，Spark程序在创建配置时死亡:

conf = HBaseConfiguration.create();
for (Iterator<Map.Entry<String, String>> it = conf.iterator(); it.hasNext(); ) {
Map.Entry<String, String> e = it.next();
System.out.println(e);
}
connection = ConnectionFactory.createConnection(conf);
admin = connection.getAdmin();

我尝试添加资源:

conf = HBaseConfiguration.create();
conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/etc/hbase/conf/hbase-site.xml"));

没有成功。

我已经注释掉hbase . configuration.create()之后的所有行，但程序还是死了。我认为问题就在那里。我没有得到有用的堆栈跟踪。司机在撞线后立即死亡。

POM:

<properties>
<spark.version>2.4.8</spark.version>
<hbase.version>1.4.13</hbase.version>
<hadoop.version>2.10.1</hadoop.version>
<jackson.version>2.13.2</jackson.version>
<!-- Maven stuff -->
<java.build.version>1.8</java.build.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bom</artifactId>
<version>2.17.103</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>4.1.77.Final</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>3.9.9.Final</version>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>${hbase.version}</version>
<scope>provided</scope>
</dependency>
<!--  AWS -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>athena</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>auth</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>opensearch</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apache-client</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.4.15</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>5.6.16</version>
</dependency>

Answer 1

删除"提供"作用域——从而包括对jar的依赖，加上添加Hbase的pom依赖(不能100%确定这是否严格必要)

<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>${hbase.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<version>${hbase.version}</version>
<type>pom</type>
</dependency>