在创建我们的 Dataproc Spark 集群时，我们传递 --properties spark:spark.jars.packages=mysql:mysql-connector-java:6.0.6到gcloud dataproc clusters create命令。
这是为了我们的 PySpark 脚本保存到 CloudSQL
显然，在创建时，这不会执行任何操作，但是在第一个spark-submit，这将尝试解决此依赖关系。
从技术上讲，它似乎解决了并下载了必要的 jar 文件，但由于发出警告，群集上的第一个任务将失败spark-submit
Exception in thread "main" java.lang.RuntimeException: [download failed: mysql#mysql-connector-java;6.0.6!mysql-connector-java.jar]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1177)
at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:298)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
完整的输出是：
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/usr/lib/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
mysql#mysql-connector-java added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found mysql#mysql-connector-java;6.0.6 in central
downloading https://repo1.maven.org/maven2/mysql/mysql-connector-java/6.0.6/mysql-connector-java-6.0.6.jar ...
:: resolution report :: resolve 527ms :: artifacts dl 214ms
:: modules in use:
mysql#mysql-connector-java;6.0.6 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 1 | 1 | 0 || 1 | 0 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
[FAILED ] mysql#mysql-connector-java;6.0.6!mysql-connector-java.jar: Downloaded file size doesn't match expected Content Length for https://repo1.maven.org/maven2/mysql/mysql-connector-java/6.0.6/mysql-connector-java-6.0.6.jar. Please retry. (212ms)
[FAILED ] mysql#mysql-connector-java;6.0.6!mysql-connector-java.jar: Downloaded file size doesn't match expected Content Length for https://repo1.maven.org/maven2/mysql/mysql-connector-java/6.0.6/mysql-connector-java-6.0.6.jar. Please retry. (212ms)
==== central: tried
https://repo1.maven.org/maven2/mysql/mysql-connector-java/6.0.6/mysql-connector-java-6.0.6.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: FAILED DOWNLOADS ::
:: ^ see resolution messages for details ^ ::
::::::::::::::::::::::::::::::::::::::::::::::
:: mysql#mysql-connector-java;6.0.6!mysql-connector-java.jar
::::::::::::::::::::::::::::::::::::::::::::::
但是，群集上的后续任务会显示此输出
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/usr/lib/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
mysql#mysql-connector-java added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found mysql#mysql-connector-java;6.0.6 in central
:: resolution report :: resolve 224ms :: artifacts dl 5ms
:: modules in use:
mysql#mysql-connector-java;6.0.6 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 0 | 0 | 0 || 1 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 1 already retrieved (0kB/7ms)
所以我的问题是：
- 原因是什么，GCP 的好人可以解决这个问题吗？
- 除了运行允许在群集启动时失败的虚拟任务之外，是否有临时解决方法？
你能如何一致地重现这个？尝试使用不同的群集设置进行重现后，我最好的理论是这可能是返回 5xx 错误的过载服务器。
就解决方法而言：
1( 从 Maven Central 下载 jar 并在提交作业时--jars选项传递它。如果您经常创建新集群，则通过初始化操作在集群上暂存此文件是可行的方法。
2(通过指向Google Maven Central镜像的属性提供备用常春藤设置文件spark.jars.ivySettings(这应该减少/消除5xx错误的几率(
请参阅此文章：https://www.infoq.com/news/2015/11/maven-central-at-google