NiFi PutHiveStreaming 处理器与 Hive:无法连接到端点

Question

有人会用Nifi 1.3.0和Hive帮助解决这个问题。我在 Hive 1.2 和 Hive 2.1.1 中遇到相同的错误。蜂巢表被分割、储存并存储为ORC格式。

分区是在 hdfs 上创建的，但数据在写入阶段失败。请检查日志如下：

[5:07 AM] papesdiop: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='mydb', table='guys', partitionVals=[dev] }
[5:13 AM] papesdiop: I get in log see next, hope it might help too:
[5:13 AM] papesdiop: Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://localhost:9083', database='mydb', table='guys', partitionVals=[dev] }
  at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578)

完整的跟踪日志：

重新。 org.apache.thrift.transport.TTransportException： null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java：132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java：86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java：429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java：318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java：219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java：69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java：3906) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java：3893) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java：1863) at sun.reflect.GeneratedMethodAccessor380.invoke(未知来源) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java：43) at java.lang.reflect.Method.invoke(Method.java：498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java：152) at com.sun.proxy.$Proxy 126.lock(未知来源) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java：573) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java：547) at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java：261) at org.apache.nifi.util.hive.HiveWriter.(蜂巢作家.java：73) at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java：46) at org.apache.nifi.processor.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java：964) at org.apache.nifi.processor.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java：875) at org.apache.nifi.processor.hive.PutHiveStreaming.lambda$null$40(PutHiveStreaming.java：676) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java：127) at org.apache.nifi.processor.hive.PutHiveStreaming.lambda$onTrigger$44(PutHiveStreaming.java：673) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java：2136) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java：2106) at org.apache.nifi.processor.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java：627) at org.apache.nifi.processor.hive.PutHiveStreaming.lambda$onTrigger$36(PutHiveStreaming.java：551) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java：114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java：184) at org.apache.nifi.processor.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java：551) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java：1120) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java：147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java：47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java：132) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java：511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java：308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java：180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java：294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java：1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java：624) at java.lang.Thread.run(Thread.java：748) 2017-09-07 06：41：31,015 调试 [Timer-4] o.a.n.processor.hive.PutHiveStreaming PutHiveStreaming[id=13ed53d2-015e-1000-c7b1-5af434c38751] 开始在所有编写器上发送检测信号 2017-09-07 06：41：31,890 信息 [计时器驱动的进程线程-3] hive.metastore 尝试使用 URI 连接到元存储 thrift://localhost:9083 2017-09-07 06：41：31,893 信息 [计时器驱动的进程线程-3] hive.metastore 连接到元存储。2017-09-07 06：41：31,911 错误 [计时器驱动的进程线程-3] o.a.n.processor.hive.PutHiveStreaming PutHiveStreaming[id=13ed53d2-015e-1000-c7b1-5af434c38751] 无法为端点创建 HiveWriter： {metaStoreUri='thrift：//localhost：9083'， database='default'， table='guys'， partitionVals=[dev] }： org.apache.nifi.util.hive.HiveWriter$ConnectFailure： 无法连接到 EndPoint {metaStoreUri='thrift：//localhost：9083'， database='default'， table='guys'， partitionVals=[dev] } org.apache.nifi.util.hive.HiveWriter$ConnectFailure： Failed connected to EndPoint {metaStoreUri='thrift：//localhost：9083'， database='default'， table='guys'， partitionVals=[dev] } at org.apache.nifi.util.hive.HiveWriter.(HiveWriter.java：79) at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java：46) at org.apache.nifi.processor.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java：964) at org.apache.nifi.processor.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java：875) at org.apache.nifi.processor.hive.PutHiveStreaming.lambda$null$40(PutHiveStreaming.java：676) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java：127) at org.apache.nifi.processor.hive.PutHiveStreaming.lambda$onTrigger$44(PutHiveStreaming.java：673) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java：2136) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java：2106) at org.apache.nifi.processor.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java：627) at org.apache.nifi.processor.hive.PutHiveStreaming.lambda$onTrigger$36(PutHiveStreaming.java：551) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java：114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java：184) at org.apache.nifi.processor.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java：551) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java：1120) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java：147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java：47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java：132) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java：511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java：308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java：180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java：294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java：1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java：624) at java.lang.Thread.run(Thread.java：748) 由：org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure：从端点获取事务批处理失败：{metaStoreUri='thrift：//localhost：9083'， database='default'， table='guys'， partitionVals=[dev] } at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java：264) at org.apache.nifi.util.hive.HiveWriter.(蜂巢作家.java：73) ...省略 24 个常见帧 由： org.apache.hive.hcatalog.streaming.TransactionError： 无法获取 lock on {metaStoreUri='thrift：//localhost：9083'， database='default'， table='guys'， partitionVals=[dev] } at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java：578) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java：547) at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java：261) ...省略 25 个常见帧 由以下原因引起：org.apache.thrift.transport.TTransport异常：null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java：132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java：86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java：429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java：318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java：219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java：69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java：3906) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java：3893) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java：1863) at sun.reflect.GeneratedMethodAccessor380.invoke(未知来源) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java：43) at java.lang.reflect.Method.invoke(Method.java：498)at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java：152) at com.sun.proxy.$Proxy 126.lock(未知来源) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java：573) ...省略了 27 个常见帧 2017-09-07 06：41：31,911 错误 [计时器驱动的进程线程-3] o.a.n.processor.hive.PutHiveStreaming PutHiveStreaming[id=13ed53d2-015e-1000-c7b1-5af434c38751] 连接到 Hive 终结点时出错：thrift://localhost:9083 的表伙计 2017-09-07 06：41：31,911 调试 [计时器驱动的进程线程-3] o.a.n.processor.hive.PutHiveStreaming PutHiveStreaming[id=13ed53d2-015e-1000-c7b1-5af434c38751] 已选择放弃其资源;不会计划在 1000 毫秒内再次运行 2017-09-07 06：41：31,912 错误 [计时器驱动的进程线程-3] o.a.n.processor.hive.PutHiveStreaming PutHiveStreaming[id=13ed53d2-015e-1000-c7b1-5af434c38751] Hive 流式处理连接/写入错误，流文件将受到惩罚并路由重试。org.apache.nifi.util.hive.HiveWriter$ConnectFailure： Failed connected to EndPoint {metaStoreUri='thrift：//localhost：9083'， database='default'， table='guys'， partitionVals=

蜂巢表

创建表mydb.guys(   firstname字符串，   lastname字符串) 分区依据 (   job字符串) 聚类依据 (   名字) 放入 10 个存储桶 行格式 塞尔德   'org.apache.hadoop.hive.ql.io.orc.OrcSerde' 存储为 ORC 位置   "hdfs://localhost:9000/user/papesdiop/guys" TBLPROPERTIES ( 'transactional'='true')

提前致谢

Answer 1

如果在写入 HDFS 期间此操作失败，则可能是您的用户没有写入目标目录的权限？如果您有来自完整堆栈跟踪的更多信息，请将其添加到您的问题中，因为它将有助于诊断问题。当我不久前遇到这个问题时，这是因为我的NiFi用户需要在目标操作系统上创建并添加到适当的HDFS组中，以便获得PutHiveStreaming写入HDFS中的ORC文件的权限。