将系统更新到 HDP 2.6.5 后出现问题。我有一个包含三个节点的集群，并尝试使用 python 启动一个简单的应用程序：
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext, SparkSession, HiveContext
sc = SparkContext()
print sc.master
带命令
/usr/bin/spark-submit
--master yarn
--deploy-mode cluster
--name 'test script'
/opt/test/youdmp/test/script.py
它说
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.yarn.proto.YarnProtos$ResourceProtoOrBuilder.getMemory()I
at org.apache.hadoop.yarn.api.records.impl.pb.ResourcePBImpl.getMemory(ResourcePBImpl.java:61)
at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:313)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:166)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1217)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1585)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:906)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
当应用程序在一个或另一个节点上启动时，我可以看到这种情况。但是第三个节点有效。
客户端部署模式说：
Traceback (most recent call last):
File "/opt/test/youdmp/test/script.py", line 3, in <module>
sc = SparkContext()
File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/context.py", line 119, in __init__
File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/context.py", line 181, in _do_init
File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/context.py", line 279, in _initialize_context
File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1428, in __call__
File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoSuchMethodError: org.apache.hadoop.yarn.proto.YarnProtos$ResourceProtoOrBuilder.getMemory()I
at org.apache.hadoop.yarn.api.records.impl.pb.ResourcePBImpl.getMemory(ResourcePBImpl.java:61)
at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:313)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:166)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
会是什么样的错误？我应该解决什么问题？
似乎 Yarn 无法在某些节点上启动应用程序大师，因为节点上的某些库是不同的。但是我无法找出哪些库，两个节点都有相似的库
在我的案例中有一些库冲突。 修复我们应该添加选项spark.driver.extraClassPath.
在我们的例子中spark.driver.extraClassPath /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hive-server2-hive2/lib/hive-hcatalog-core.jar