我正在尝试连接(PySpark+Snowflake)，但一直收到错误。
我使用的PySpark 3.1、JDBC和Spark_Snowflake jar文件也放在Classpath中。不确定我为什么会出现以下错误。
代码：
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark import SparkConf, SparkContext
sc = SparkContext("local", "Test App")
spark = SQLContext(sc)
spark_conf = SparkConf().setMaster('local').setAppName('Testing Spark SF')
sfOptions = {
"sfURL" : "<account_identifier>.snowflakecomputing.com",
"sfUser" : "<user_name>",
"sfPassword" : "<password>",
"sfDatabase" : "<database>",
"sfSchema" : "<schema>",
"sfWarehouse" : "<warehouse>"
}
SNOWFLAKE_SOURCE_NAME = "net.snowflake.spark.snowflake"
df = spark.read.format(SNOWFLAKE_SOURCE_NAME)
.options(**sfOptions)
.option("query", "select 1 as my_num union all select 2 as my_num")
.load()
df.show()
错误：
Py4JJava错误：调用o37.load时出错。：net.snowflake.client.jdbc.SnowflakeSQL异常：jdbc驱动程序遇到通信错误。消息：遇到的异常HTTP请求：sun.security.validator.ValidatorException：不受信任找到证书。在net.snowlake.client.jdbc.RestRequest.execute(RestRequest.java:284)在net.snowlake.client.core.HttpUtil.executeRequestInternal(HttpUtil.java:639)在net.snowlake.client.core.HttpUtil.executeRequest(HttpUtil.java:584)在net.snowlake.client.core.HttpUtil.executeGeneralRequest(HttpUtil.java:551)在net.snowlake.client.core.SessionUtil.newSession(SessionUtil.java:587)在net.s雪花.client.core.SessionUtil.openSession(SessionUtil.java:285)位于的net.snowlake.client.core.SFSession.open(SFSession.java:446)net.snowlake.client.jdbc.DefaultSFConnectionHandler.initialize(DefaultSFConnectionHandler.java:104)在net.snowlake.client.jdbc.DefaultSFConnectionHandler.initializeConnection(DefaultSFConnectionHandler.java:79)在net.snowflake.client.jdbc.SnowflakeConnectionV1.initConnectionWithImpl(SnowflakeConnectionV1.java:116)在net.snowlake.client.jdbc.SnowflakeConnection V1.(SnowflakeConnectionV1.java:96)在net.snowlake.client.jdbc.SnowflakeDriver.connect(SnowflakeDriver.java:172)位于java.sql.DriverManager.getConnection(DriverManager.java:664)java.sql.DriverManager.getConnection(DriverManager.java:208)位于net.snowlake.spark.snowflake.JDBCWrapper.getConnector(SnowflakeJDBCWraper.scala:209)在net.snowlake.spark.snowflake.SnowflakeRelation.$anonfun$schema$1(SnowflakeLelation.scala:60)在net.snowlake.spark.snowflake.SnowflakeRelation$$Lambda$866/22415031.apply(未知Source)。Option.getOrElse(Option.scala:189)net.snowlake.spark.snowflake.SnowflakeRelation.schema$lzycompute(SnowflakeLelation.scala:57)在net.snowlake.spark.snowflake.SnowflakeRelation.schema(SnowflakeLelation.scala:56)在org.apache.spark.sql.exexecution.datasources.DataSource.resolveRelation(DataSource.scala:449)在org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)在org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)在org.apache.spark.sql.DataFrameReader$$Lambda$858/5135046.apply(未知Source)。Option.getOrElse(Option.scala:189)org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)在org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:225)位于的sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在java.lang.reflect.Method.ioke(Method.java:497)位于的py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)py4j.reflection.ReflectionEngine.reinvoke(ReflectionEngine.java:357)在py4j.Gateway。invoke(Gateway。java:282)py4j.commands.AbstractCommand.invokeMethod(AbstractCmd.java:132)在py4j.commands.CallCommand.execute(CallCommand.java:79)py4j.GatewayConnection.run(GatewayConnection.java:238)位于java.lang.Thread.run(Thread.java:745)由以下原因引起：javax.net.ssl.ssl握手异常：sun.security.validator.ValidatorException：没有可信证书位于sun.security.ssl.Alerts.getSSLException(Alerts.java:198)在sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1958)位于sun.security.ssl.Handshaker.totalSE(Handshaker.java:322)sun.security.ssl.Handshaker.totalSE(Handshaker.java:316)位于sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1526)在sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:215)在sun.security.ssl.Handshaker.processLoop(Handshaker.java:1024)位于sun.security.ssl.Handshaker.process_record(Handshaker.java:954)在sun.security.ssl.SLSocketImpl.readRecord(SSLSocketImpl.java:1065)在sun.security.ssl.SLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1384)在sun.security.ssl.SLSocketImpl.startHandshake(SSLSocketImpl.java:1412)在sun.security.ssl.SLSocketImpl.startHandshake(SSLSocketImpl.java:1396)在net.snowlake.client.jdbc.internal.apache.http.con.sl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
很明显，SSL证书存在问题。您可以临时覆盖它。
sfOptions = {
...
"sfSSL" : "false",
}
但是，您可以检查是否通过代理访问Snowflake。您需要导入代理证书，并将其包含在您的证书中。正在运行的java版本的cacerts的默认位置。您可以在lib/security下的java主目录中找到它。
keytool -import -trustcacerts -alias cert_ssl -file proxy.cer -noprompt -storepass changeit -keystore cacerts