无法在Jupyter Notebook上创建火花上下文

Question

有人能帮我在创建Spark Context时出现以下错误吗？我无法创建，因为我得到以下错误

from pyspark import SparkConf,SparkContext
conf=SparkConf().setMaster("localhost")
sc=SparkContext(conf = conf)

ExceptionTraceback (most recent call last)
<ipython-input-7-3d97d6624879> in <module>()
----> 1 sc=SparkContext(conf = conf)
/home/cloudera/anaconda2/lib/python2.7/site-packages/pyspark/context.pyc in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
113         """
114         self._callsite = first_spark_call() or CallSite(None, None, None)
--> 115         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
116         try:
117             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
/home/cloudera/anaconda2/lib/python2.7/site-packages/pyspark/context.pyc in _ensure_initialized(cls, instance, gateway, conf)
296         with SparkContext._lock:
297             if not SparkContext._gateway:
--> 298                 SparkContext._gateway = gateway or launch_gateway(conf)
299                 SparkContext._jvm = SparkContext._gateway.jvm
300 
/home/cloudera/anaconda2/lib/python2.7/site-packages/pyspark/java_gateway.pyc in launch_gateway(conf)
92 
93             if not os.path.isfile(conn_info_file):
---> 94                 raise Exception("Java gateway process exited before sending its port number")
95 
96             with open(conn_info_file, "rb") 
Exception: Java gateway process exited before sending its port number

Answer 1

当您的PySpark版本与您设置的Spark版本不匹配时，可能会发生这种情况。(由SPARK_HOME变量指示(-我记得这是很久以前的问题。

在您的终端上，尝试进行

pip freeze|grep pyspark

然后

spark-shell --version

确保版本匹配。如果是，请重新启动内核，然后重试。希望这能有所帮助！