使用Spark 2.1.0和Spark-Cassandra-Connector 2.0.0.0-RC1-S_2.11。在与其他转换相同的功能内完成时,调用rdd.saveToCassandra("keyspace", "table", SomeColumns("A", "B"))
正常工作。但是,在同一对象上定义一种方法:
def saveToCassandra(rdd: RDD[_], keyspace: String, table: String, columns: SomeColumns) = {
rdd.saveToCassandra(keyspace, table, columns)
}
并用saveToCassandra(rdd, "keyspace", "table", SomeColumns("A", "B"))
调用它失败:
Exception in thread "main" scala.ScalaReflectionException: <none> is not a term
at scala.reflect.api.Symbols$SymbolApi$class.asTerm(Symbols.scala:199)
at scala.reflect.internal.Symbols$SymbolContextApiImpl.asTerm(Symbols.scala:84)
at com.datastax.spark.connector.util.Reflect$.methodSymbol(Reflect.scala:12)
at com.datastax.spark.connector.util.ReflectionUtil$.constructorParams(ReflectionUtil.scala:63)
at com.datastax.spark.connector.mapper.DefaultColumnMapper.<init>(DefaultColumnMapper.scala:45)
at com.datastax.spark.connector.mapper.LowPriorityColumnMapper$class.defaultColumnMapper(ColumnMapper.scala:51)
at com.datastax.spark.connector.mapper.ColumnMapper$.defaultColumnMapper(ColumnMapper.scala:55)
用rdd的类型标签定义辅助方法,例如rdd: RDD[(String, String)]
而不是RDD[_]
解决了问题。我想这与DataStax连接器中的反射代码以及它从编译时类型中输入运行时类型的方式有关。显然将下划线作为任何类型的擦除类型信息进行反思。我有点新来Scala,不知道反射在Scala中的工作原理以及是否需要这种行为。