Infrorror:无法导入名称SQLContext

Question

我正在使用pyspark读取一些CSV数据以引发数据框架。

我尝试导入pyspark模块，如下所示：

from pyspark.sql import sqlContext

为什么我会收到以下错误？如何修复它？

Importerror：无法导入名称sqlcontext

我正在使用Python 2.7和Spark 2.0.1

Answer 1

这可能是因为您没有正确的python pat设置。我发现以下功能在配置Python环境时很有用。

def configure_spark(spark_home=None, pyspark_python=None, conf_dir=None):
    """Configures the Python path for importing pyspark
    Sets the SPARK_HOME and PYSPARK_PYTHON environment variables and modifies
    the Python PATH so the pyspark package can be imported.
    Args:
        spark_home (str): Path of SPARK_HOME. Defaults to SPARK_HOME module
            variable.
        pyspark_python (str): Path to Python binary to use in PySpark. Defaults
            to the currently executing Python binary.
        conf_dir (str): Path to configuration directory
    """
    # Set the configuration directory with some basic sanity checks:
    if conf_dir:
        if not os.path.isdir(conf_dir):
            raise OSError("Spark config directory not found: %s" % conf_dir)
        expected_conf = {'spark-env.sh', 'spark-defaults.conf'}
        found_conf = expected_conf - set(os.listdir(conf_dir))
        if found_conf:
            warnings.warn("Some configuration files were not found: %s" % found_conf)
        os.environ['SPARK_CONF_DIR'] = conf_dir
    spark_home = spark_home or SPARK_HOME
    os.environ['SPARK_HOME'] = spark_home
    if not os.path.isdir(spark_home):
        raise OSError("Specified SPARK_HOME is not a valid directory: %s" % spark_home)
    # Add the PySpark directories to the Python path:
    libs = glob(os.path.join(spark_home, 'python', 'lib', '*.zip'))
    if len(libs) < 2:
        raise OSError("Pyspark libraries not found in %s" % spark_home)
    for lib in libs:
        sys.path.insert(1, lib)
    # If PYSPARK_PYTHON isn't specified, use currently running Python binary:
    pyspark_python = pyspark_python or sys.executable
    os.environ['PYSPARK_PYTHON'] = pyspark_python

Answer 2

您也可以尝试使用pyspark，我在jupyter笔记本中遇到了相同的问题，您可以使用

来解决

from pyspark import SQLContext