即使存在JAR文件，也无法在pyspark中实例化GoogleHadoopFileSystem &g

Question

相同的代码在Linux Ubuntu上使用相同的jar文件运行完全正常。我的spark是3.1.2,hadoop是3.2。

val = df.write.format('bigquery')        #df is a spark.dataframe
.mode(mode) 
.option("credentialsFile", "creds.json") 
.option('table', table) 
.option("temporaryGcsBucket", bucket) 
.save()

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/09/17 07:41:50 WARN FileSystem: Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
21/09/17 07:41:50 WARN FileSystem: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkState(ZLjava/lang/String;J)V
21/09/17 07:41:50 WARN FileSystem: Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
21/09/17 07:41:50 WARN FileSystem: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
Traceback (most recent call last):
File "c:sparktestmain.py", line 158, in <module>
val = df.write.format('bigquery') 
File "c:sparktestvnenvlibsite-packagespysparksqlreadwriter.py", line 828, in save
self._jwrite.save()
File "c:sparktestvnenvlibsite-packagespy4jjava_gateway.py", line 1304, in __call__
return_value = get_return_value(
File "c:sparktestvnenvlibsite-packagespysparksqlutils.py", line 128, in deco
return f(*a, **kw)
File "c:sparktestvnenvlibsite-packagespy4jprotocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o50.save.
: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem

Answer 1

spark jars文件夹中的旧jar没有正确实例化。我不得不从maven仓库中删除它们并添加新的。下面是我使用的代码:

spark = SparkSession 
.builder 
.appName(appName) 
.config(conf=spark_conf) 
.config('spark.jars.packages', 'com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.22.0') 
.getOrCreate()