如何将CUDA与vaex(一个Python库)一起使用



我的代码如下:

df['O_ID'] = (df.apply(get_match_id, arguments=[df['pickup_longitude'], df['pickup_latitude']])).jit_cuda()

当我第一次使用这个函数----jit_cuda((时,出现了一个错误"没有命名为"cupy"的模块;

但是,当我安装了cupy-cuda101(适应我的CUDA版本(我得到一个新的错误

Traceback (most recent call last):
File "F:Anaconda3libsite-packagesvaexdataframe.py", line 3580, in table_part
values[name] = df.evaluate(name)
File "F:Anaconda3libsite-packagesvaexdataframe.py", line 2616, in evaluate
return self._evaluate_implementation(expression, i1=i1, i2=i2, out=out, selection=selection, filtered=filtered, internal=internal, parallel=parallel, chunk_size=chunk_size)
File "F:Anaconda3libsite-packagesvaexdataframe.py", line 5352, in _evaluate_implementation
dtypes[expression] = df.data_type(expression, internal=False)
File "F:Anaconda3libsite-packagesvaexdataframe.py", line 1998, in data_type
data = self.evaluate(expression, 0, 1, filtered=True, internal=True, parallel=False)
File "F:Anaconda3libsite-packagesvaexdataframe.py", line 2616, in evaluate
return self._evaluate_implementation(expression, i1=i1, i2=i2, out=out, selection=selection, filtered=filtered, internal=internal, parallel=parallel, chunk_size=chunk_size)
File "F:Anaconda3libsite-packagesvaexdataframe.py", line 5427, in _evaluate_implementation
value = scope.evaluate(expression)
File "F:Anaconda3libsite-packagesvaexscopes.py", line 97, in evaluate
result = self[expression]
File "F:Anaconda3libsite-packagesvaexscopes.py", line 139, in __getitem__
self.values[variable] = self.evaluate(expression)  # , out=self.buffers[variable])
File "F:Anaconda3libsite-packagesvaexscopes.py", line 103, in evaluate
result = eval(expression, expression_namespace, self)
File "<string>", line 1, in <module>
File "F:Anaconda3libsite-packagesvaexexpression.py", line 1073, in __call__
return self.f(*args, **kwargs)
File "F:Anaconda3libsite-packagesvaexexpression.py", line 1120, in wrapper
return cupy.asnumpy(func(*args))
File "cupycorefusion.pyx", line 905, in cupy.core.fusion.Fusion.__call__
File "cupycorefusion.pyx", line 754, in cupy.core.fusion._FusionHistory.get_fusion
File "<string>", line 6, in f
NameError: name 'lambda_function_1' is not defined

我应该如何解决它?

我的理解是,vaex中的即时编译只适用于虚拟列,或者主要使用numpy方法或纯python算法通过各种算术运算计算的表达式/列。

当使用apply时,一个函数可能非常抽象,基本上是你想要的任何函数,所以它可能不可能被编译。

如果您可以使用numpy表达式重写.apply函数,那么您可能可以使用jit_cuda方法来加速它。Vaex无论如何都不建议使用.apply,因为它很难并行化,应该使用";最后的手段";在某种程度上。

来源:https://vaex.io/docs/tutorial.html#Just-实时编译

最新更新