在Windows投掷错误上创建HiveContext

Question

我正在尝试创建HiveContext，但其投掷错误。

只是因为我没有winutils.exe吗？或我该如何解决此问题？

我要创建HiveContext的原因是我打算使用collect_set函数，该功能是在Hive中创建的。

错误日志：

16/11/23 14:12:09 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/11/23 14:12:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/11/23 14:12:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/11/23 14:12:14 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/11/23 14:12:14 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/11/23 14:12:15 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/11/23 14:12:15 INFO ObjectStore: Initialized ObjectStore
16/11/23 14:12:15 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/11/23 14:12:16 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/11/23 14:12:16 WARN : Your hostname, INNR90GLH5G resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:c0a8:1584%net6, but we couldn't find any external IP address!
16/11/23 14:12:17 INFO HiveMetaStore: Added admin role in metastore
16/11/23 14:12:17 INFO HiveMetaStore: Added public role in metastore
16/11/23 14:12:17 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/11/23 14:12:18 INFO HiveMetaStore: 0: get_all_databases
16/11/23 14:12:18 INFO audit: ugi=1554161   ip=unknown-ip-addr  cmd=get_all_databases   
16/11/23 14:12:18 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/11/23 14:12:18 INFO audit: ugi=1554161   ip=unknown-ip-addr  cmd=get_functions: db=default pat=* 
16/11/23 14:12:18 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
Exception in thread "main" java.lang.RuntimeException: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
    at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
    at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
    at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
    at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)
    at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)
    at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
    at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
    at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
    at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
    at com.scb.cnc.payments.CockpitChannelStatus$.main(CockpitChannelStatus.scala:13)
    at com.scb.cnc.payments.CockpitChannelStatus.main(CockpitChannelStatus.scala)
Caused by: java.lang.NullPointerException
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
    at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:572)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:547)
    at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
    ... 12 more

Answer 1

如果问题缺乏winutils.exe，尽管有hadoop并在您的工作站上发生火花，请尝试本文以解决您的问题

Spark