我正在尝试自学Scala,并使用IntelliJ IDEA作为我的IDE。我已经启动了IntelliJ的sbt shell,运行console
,然后输入以下内容:
import org.apache.spark.SparkConf
import org.apache.spark.sql.{DataFrame, SparkSession}
import java.time.LocalDate
object DataFrameExtensions {
implicit class DataFrameExtensions(df: DataFrame){
def featuresGroup1(groupBy: Seq[String], asAt: LocalDate): DataFrame = {df}
def featuresGroup2(groupBy: Seq[String], asAt: LocalDate): DataFrame = {df}
}
}
import DataFrameExtensions._
val spark = SparkSession.builder().config(new SparkConf().setMaster("local[*]")).enableHiveSupport().getOrCreate()
import spark.implicits._
val df = Seq((8, "bat"),(64, "mouse"),(-27, "horse")).toDF("number", "word")
val groupBy = Seq("a","b")
val asAt = LocalDate.now()
val dataFrames = Seq(df.featuresGroup1(groupBy, asAt),df.featuresGroup2(groupBy, asAt))
它在最后一行失败,并显示:
scala> val dataFrames = Seq(df.featuresGroup1(groupBy, asAt),df.featuresGroup2(groupBy, asAt))
<console>:25: error: value featuresGroup1 is not a member of
org.apache.spark.sql.DataFrame
val dataFrames = Seq(df.featuresGroup1(groupBy, asAt),df.featuresGroup2(groupBy, asAt))
^
<console>:25: error: value featuresGroup2 is not a member of org.apache.spark.sql.DataFrame
val dataFrames = Seq(df.featuresGroup1(groupBy, asAt),df.featuresGroup2(groupBy, asAt))
^
我已经从其他地方(我知道它有效(逐字复制了代码,所以我不知道为什么这不起作用。为什么在我的隐式类中定义的函数不能用作DataFrame
上的函数?
似乎您需要将implicit class DataFrameExtensions
重命名为其他名称,因为有一个具有相同名称的object
。我想编译器在使用时对定位implicit class
感到困惑
import DataFrameExtensions._
我将其重命名为下面,它现在可以工作了
implicit class FeatureGroupImplicits(df: DataFrame){
def featuresGroup1(groupBy: Seq[String], asAt: LocalDate): DataFrame = {df}
def featuresGroup2(groupBy: Seq[String], asAt: LocalDate): DataFrame = {df}
}
(问题已编辑,答案已过时,以后可能会编辑(
这里:
val groupBy = Seq('a','b')
产生Seq[Char]
,而不是Seq[String]
..由于DataFrame
上没有方法featuresGroup1(s: Seq[Char], d: LocalDate)
,scalac 将尝试找到一个隐式转换为具有此方法的类。但是,只有
def featuresGroup1(groupBy: Seq[String], asAt: LocalDate): DataFrame
所以Seq[Char]
和Seq[String]
不适合在一起,编译器甚至没有报告类似的东西
方法
featuresGroup1
的参数类型错误 预期:序列[字符串] 收到: 塞克[字符]
因为它根本没有看到与此方法的任何联系。
请改用Seq("a", "b")
。