java.lang.ClassCastException:java.util.ArrayList 不能在 pyspark

Question

我正在尝试使用以下代码使用 pyspark 2.10 创建一个数据透视表。

pivot = dataset_join.groupby('DCLNTSOTP').pivot('YYYYMM').sum(['REV_NET_CLIENT_REVENUE'])

我收到以下错误消息：

java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String

完整的堆栈轨道在这里。

 Py4JJavaError                             Traceback (most recent call last)
<command-2998756010413502> in <module>()
----> 1 pivot = dataset_join.groupby("DCLNTSOTP").pivot("YYYYMM").sum(['REV_NET_CLIENT_REVENUE'])
/databricks/spark/python/pyspark/sql/group.py in _api(self, *cols)
     39     def _api(self, *cols):
     40         name = f.__name__
---> 41         jdf = getattr(self._jgd, name)(_to_seq(self.sql_ctx._sc, cols))
     42         return DataFrame(jdf, self.sql_ctx)
     43     _api.__name__ = f.__name__
/databricks/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1158         answer = self.gateway_client.send_command(command)
   1159         return_value = get_return_value(
-> 1160             answer, self.gateway_client, self.target_id, self.name)
   1161 
   1162         for temp_arg in temp_args:
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()
/databricks/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    318                 raise Py4JJavaError(
    319                     "An error occurred while calling {0}{1}{2}.n".
--> 320                     format(target_id, ".", name), value)
    321             else:
    322                 raise Py4JError(
Py4JJavaError: An error occurred while calling o12780.sum.
: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String
    at org.apache.spark.sql.RelationalGroupedDataset$$anonfun$3.apply(RelationalGroupedDataset.scala:101)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at org.apache.spark.sql.RelationalGroupedDataset.aggregateNumericColumns(RelationalGroupedDataset.scala:101)
    at org.apache.spark.sql.RelationalGroupedDataset.sum(RelationalGroupedDataset.scala:299)
    at sun.reflect.GeneratedMethodAccessor592.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
    at py4j.Gateway.invoke(Gateway.java:295)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:226)
    at java.lang.Thread.run(Thread.java:748)

在我的数据中，"DCLNTSOTP"是一个字符串，YYYYMM"是一个整数，REV_NET_CLIENT_REVENUE是一个小数。

任何帮助将不胜感激。

Answer 1

我最近在同事的项目中观察到了同样的问题。删除REV_NET_CLIENT_REVENUE ？ 还要确保REV_NET_CLIENT_REVENUE是数字而不是字符串。