加载ML PySpark模型失败

Question

我有几个无法加载的回归模型。这是Spark init:

from pyspark.sql import SparkSession, SQLContext
from pyspark.ml.regression import DecisionTreeRegressor
spark = SparkSession.builder 
.appName("Linear Regression Model") 
.config('spark.executor.cores','2') 
.config("spark.executor.memory", "5gb") 
.master("local[*]") 
.getOrCreate() 
sc = spark.sparkContext

以下是模型拟合并成功保存：

# Decision Tree Regression
decisionTree = DecisionTreeRegressor(featuresCol = "Features", labelCol = "SalePrice", maxDepth = 15, maxBins = 32)
decisionTreeModel = decisionTree.fit(train_vector)
import os
decisionTreeModel.save(os.path.join(".", 'decisionTreeModel'))

但当我加载回来时：

persistedModel = DecisionTreeRegressor.load("decisionTreeModel")

我得到这个错误：

Py4JJavaError: An error occurred while calling o1201.load.
: java.lang.NoSuchMethodException: org.apache.spark.ml.regression.DecisionTreeRegressionModel.<init>(java.lang.String)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getConstructor(Class.java:1825)
at org.apache.spark.ml.util.DefaultParamsReader.load(ReadWrite.scala:468)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

有人知道如何加载PySpark模型吗？

Answer 1

错误消息帮助不大，但我认为加载回模型的正确方法是调用模型的load方法，而不是估计器的方法。模型已经拟合到数据中，这与估计器不同，后者只包含设置/参数，但没有拟合。

所以你可以试试这个：

from pyspark.ml.regression import DecisionTreeRegressionModel
persistedModel = DecisionTreeRegressionModel.load("decisionTreeModel")

供您参考，以下是加载估计器与加载模型的比较：

from pyspark.ml.regression import DecisionTreeRegressor, DecisionTreeRegressionModel
decisionTree = DecisionTreeRegressor(featuresCol = "Features", labelCol = "SalePrice", maxDepth = 15, maxBins = 32)
decisionTree.save('tree')
persistedEstimator = DecisionTreeRegressor.load('tree')
decisionTreeModel = decisionTree.fit(df)
decisionTreeModel.save('model')
persistedModel = DecisionTreeRegressionModel.load('model')