小贝子编程

为什么找不到Presto自定义UDAF中的返回类型



自定义函数我制作的内容在emr上不起作用。

我想创建一个只需返回42的简单UDAF。

首先，我的自定义功能我写了一个简单的功能，但在presto中不起作用。

presto-cli中的错误正在遵循：

presto> select answer_to_life('the universe');
Query 20180324_120433_00000_7n6s6 failed: answer_to_life(varchar):bigint not found
com.facebook.presto.spi.PrestoException: answer_to_life(varchar):bigint not found
        at com.facebook.presto.metadata.FunctionRegistry.doGetSpecializedFunctionKey(FunctionRegistry.java:972)
        at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:146)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
        at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
        at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:5153)
        at com.facebook.presto.metadata.FunctionRegistry.getSpecializedFunctionKey(FunctionRegistry.java:898)
        at com.facebook.presto.metadata.FunctionRegistry.getAggregateFunctionImplementation(FunctionRegistry.java:875)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.buildAccumulatorFactory(LocalExecutionPlanner.java:1973)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.planGlobalAggregation(LocalExecutionPlanner.java:1984)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:955)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:1919)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:196)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:952)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:638)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:393)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:324)
        at com.facebook.presto.execution.SqlTaskExecution.<init>(SqlTaskExecution.java:161)
        at com.facebook.presto.execution.SqlTaskExecution.createSqlTaskExecution(SqlTaskExecution.java:121)
        at com.facebook.presto.execution.SqlTaskExecutionFactory.create(SqlTaskExecutionFactory.java:71)
        at com.facebook.presto.execution.SqlTask.updateTask(SqlTask.java:340)
        at com.facebook.presto.execution.SqlTaskManager.updateTask(SqlTaskManager.java:321)
        at com.facebook.presto.server.TaskResource.createOrUpdateTask(TaskResource.java:128)
        at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
        at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:454)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:564)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
        at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
        at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
        at java.lang.Thread.run(Thread.java:748)

聚合功能的代码正在遵循。

我引用了此Presto代码

package sample;  
import com.facebook.presto.spi.block.BlockBuilder;  
import com.facebook.presto.spi.function.*;  
import com.facebook.presto.spi.type.BigintType;  
import com.facebook.presto.spi.type.StandardTypes;  
import io.airlift.slice.Slice;  
@AggregationFunction("answer_to_life")  
public final class AnswerToLife {  
    private AnswerToLife() {  
    }  
   @InputFunction  
   public static void input(@AggregationState NullState state, @SqlType(StandardTypes.VARCHAR) Slice value) {  
    }  
   @CombineFunction  
   public static void combine(@AggregationState NullState state, @AggregationState NullState other) {  
    }  
    @OutputFunction(StandardTypes.BIGINT)  
    public static void output(@AggregationState NullState state, BlockBuilder out) {  
        BigintType.BIGINT.writeLong(out, 42);  
   }  
}

详细代码在这里(https://github.com/asari-mtr/presto-udaf/tree/stackoverflow(。

我所部署的结构是follwoing：

$ ls -1 /usr/lib/presto/plugin/my-udaf/
commons-codec-1.4.jar
guava-21.0.jar
hive-udf-1.0-SNAPSHOT.jar
presto-array-0.197.jar
stats-0.155.jar
presto-udaf-1.0-SNAPSHOT.jar

我使用emr-5.12.0(Presto 0.188(

谢谢您的时间。

编辑1

jar文件中的列表。

% jar -tf target/presto-udaf-1.0-SNAPSHOT.jar 
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
META-INF/
META-INF/MANIFEST.MF
META-INF/services/
sample/
META-INF/services/com.facebook.presto.spi.Plugin
sample/AnswerToLife.class
sample/AnswerToLifePlugin.class
sample/NullState.class
META-INF/maven/
META-INF/maven/sample/
META-INF/maven/sample/presto-udaf/
META-INF/maven/sample/presto-udaf/pom.xml
META-INF/maven/sample/presto-udaf/pom.properties

sever.log 

$ grep answer -i  /mnt/var/log/presto/server.log
2018-03-26T06:37:14.213Z        INFO    main    com.facebook.presto.server.PluginManager        Installing sample.AnswerToLifePlugin
2018-03-26T06:37:14.214Z        INFO    main    com.facebook.presto.server.PluginManager        Registering functions from sample.AnswerToLife

并执行show funcstions 

presto> show functions;
            Function             |                     Return Type                     |                                 Argument Types
---------------------------------+-----------------------------------------------------+------------------------------------------------------------------------------
 ST_Area                         | double                                              | Geometry
 ST_AsText                       | varchar                                             | Geometry
 ...
 answer_to_life                  | bigint                                              | varchar
 ...

编辑2

删除src/main/resources/META-INF/services/com.facebook.presto.spi.Plugin

将 <packaging>presto-plugin</packaging>添加到pom.xml

https://github.com/asari-mtr/presto-udaf/commit/f2a2ddbf039e08f418b378a78a7ead511020E98A3B

我部署了Maven在/usr/lib/presto/plugin/下进行的ZIP。

但是，错误的内容不会改变。

编辑3

我从Mac上获得了Github(分支0.188(并建造Presto的来源。

当我们将上述UDAF放在其presto上时，它可以很好地工作。

也许在EMR上的Presto安装过程中存在错误。

已解决！

自定义插件不起作用，因为它不是插件的创建，但是在EMR上部署的一种方式是错误的。

由于我没有在执行的过程中部署到所有节点，因此我通过将以下外壳注册为bootstrap操作确认了该操作。

#/bin/bash
aws s3 cp s3://hogehoge/presto-udaf-1.0-SNAPSHOT.zip /tmp/
sudo mkdir -p /usr/lib/presto/plugin/
sudo unzip -d /usr/lib/presto/plugin/ /tmp/presto-udaf-1.0-SNAPSHOT.zip
  • 10.1。SPI概述 - Presto 0.198文档

必须在Presto群集中的所有节点上安装插件 (协调员和工人(。

  • 创建bootstrap操作以安装其他软件-Amazon EMR

您可以使用bootstrap操作将对象从Amazon S3复制到每个 在安装应用程序之前，在集群中的节点。AWS CLI 安装在群集的每个节点上，因此您的引导程序可以 致电AWS CLI命令。

从您写的内容中，有两件事缺少。

  • 确保您在pom.xml中使用Maven包装presto-plugin
  • 还实现com.facebook.presto.spi.Plugin，在getFunctions方法中将返回您的功能。

查看presto-geospatial PRESTO模块，请参见com.facebook.presto.plugin.geospatial.GeoPlugin及其pom.xml作为参考。

相关内容

  • 没有找到相关文章

最新更新


热门标签：

javascript python java c# php android html jquery c++ css ios sql mysql arrays asp.net json python-3.x ruby-on-rails .net sql-server django objective-c excel regex ruby linux ajax iphone xml vba spring asp.net-mvc database wordpress string postgresql wpf windows xcode bash git oracle list vb.net multithreading eclipse algorithm macos powershell visual-studio image forms numpy scala function api selenium