我正在使用Spark-Submit，当我导入包'pandas'时遇到错误。
从笔记本上，我可以毫无问题地导入熊猫。
但是当我通过火花提交提交以下文件时：
from pymongo import MongoClient
import pandas as pd
mongo_url = "mongodb://..."
LOGDB = MongoClient(mongo_url).schedule.log_test
LOGDB.insert({'m': 'Worked Spark Submit Test'})
我收到以下错误：
stderr_1461683917691631694文件：
Spark Command: /usr/local/src/spark160master/ibm-java-x86_64-80//bin/java -cp /usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-launcher_2.10-1.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-network-shuffle_2.10-1.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/gson-2.2.4.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/guava-14.0.1.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/Java-WebSocket-1.3.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-ego_2.10-1.6.0.jar:/usr/local/src/spark160master/spark/profile/batch/:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/spark-assembly-1.6.0-hadoop2.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-api-jdo-3.2.6.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-core-3.2.10.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-rdbms-3.2.9.jar:/usr/local/src/data-connectors-1.4.1/* -Dspark.eventLog.enabled=true -Dspark.master=spark://yp-spark-dal09-env5-0005:7083 -Dspark.executor.memory=1024m -Dspark.executor.extraLibraryPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.driver.extraClassPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.driver.extraLibraryPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.app.name=test_submit.py -Dspark.executor.extraClassPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.files=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py -Dspark.eventLog.dir=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/events -Dspark.files.useFetchCache=false -Dspark.shuffle.service.port=7340 -Xms512m -Xmx512m org.apache.spark.deploy.ego.EGOClusterDriverWrapper {{WORKER_URL}} /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py org.apache.spark.deploy.PythonRunner --primary-py-file test_submit.py
========================================
log4j:ERROR Could not find value for key log4j.appender.FILE
log4j:ERROR Could not instantiate appender named "FILE".
16/04/26 10:18:43 INFO deploy.ego.EGOClusterDriverWrapper: Registered signal handlers for [TERM, HUP, INT]
16/04/26 10:18:44 WARN hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/04/26 10:18:44 INFO apache.spark.SecurityManager: Changing view acls to: s270-18029e40ba8077-05540e1c503a
16/04/26 10:18:44 INFO apache.spark.SecurityManager: Changing modify acls to: s270-18029e40ba8077-05540e1c503a
16/04/26 10:18:44 INFO apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(s270-18029e40ba8077-05540e1c503a); users with modify permissions: Set(s270-18029e40ba8077-05540e1c503a)
16/04/26 10:18:45 INFO spark.util.Utils: Successfully started service 'EGOClusterDriverWrapper-driver-20160426101839-0010-41cca182-f2d6-408c-81c0-ec288530f82e' on port 55398.
16/04/26 10:18:45 INFO apache.spark.SecurityManager: Changing view acls to: s270-18029e40ba8077-05540e1c503a
16/04/26 10:18:45 INFO apache.spark.SecurityManager: Changing modify acls to: s270-18029e40ba8077-05540e1c503a
16/04/26 10:18:45 INFO apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(s270-18029e40ba8077-05540e1c503a); users with modify permissions: Set(s270-18029e40ba8077-05540e1c503a)
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Fetching file from /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py to /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py
16/04/26 10:18:45 INFO spark.util.Utils: Copying /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py to /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Starting the user JAR in a separate Thread
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Waiting for spark context initialization ... 0
16/04/26 10:18:46 INFO deploy.ego.EGOClusterDriverWrapper: Final app status: 1, exitCode: 15, (reason: User class threw exception: User application exited with 1)
16/04/26 10:18:46 INFO deploy.ego.EGOClusterDriverWrapper: Sending driver program state to master
Exception in thread "Driver" org.apache.spark.SparkUserAppException: User application exited with 1
at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:88)
at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:507)
at org.apache.spark.deploy.ego.EGOClusterDriverWrapper$$anon$3.run(EGOClusterDriverWrapper.scala:434)
stdout_1461683917691631694
no extra config
load default config from : /usr/local/src/spark160master/spark/profile/batch/
/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsUTF8String
Traceback (most recent call last):
File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py", line 19, in <module>
import pandas as pd
File "/usr/lib64/python2.7/site-packages/pandas-0.14.1-py2.7-linux-x86_64.egg/pandas/__init__.py", line 6, in <module>
from . import hashtable, tslib, lib
File "numpy.pxd", line 157, in init pandas.hashtable (pandas/hashtable.c:22150)
File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/__init__.py", line 180, in <module>
from . import add_newdocs
File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>
from .type_check import *
File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/__init__.py", line 14, in <module>
from . import multiarray
ImportError: /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsUTF8String
我做错了什么？
我无法重现您的错误，因为我没有您的整个代码 Spark-submit 作业代码。
是否正在初始化 Spark 上下文？
这是我有效的代码：-
from pymongo import MongoClient
import pandas as pd
from pyspark import SparkContext
if __name__ == "__main__":
sc = SparkContext(appName="MongoDB CON TEST")
print("Hello Spark Demo for Mongodb connection")
print("Begining MongoCon")
mongo_url = "mongodb://ds045252.mlab.com:45252"
connection = MongoClient(mongo_url)
db = connection["samplemongodb"]
# MongoLab has user authentication
db.authenticate("username", "password")
db.Photos.insert({'m': 'Worked Spark Submit Test Real test'})
print("Insert worked")
sc.stop()
请在 github 中查看带有缩进的完整代码：-https://github.com/charles2588/sparkjobs/blob/master/mongodbtest.py
提交以下工作：-
./spark-submit.sh --vcap credentials/vcap.json --deploy-mode cluster --master https://169.54.219.20 mongodbtest.py
谢谢
查尔斯。
笔记本预装了Pandas和NumPy。您的堆栈跟踪显示 spark-submit 正在尝试从 .local 加载 NumPy，因此这是您自己安装的版本。在 Spark-submit 环境中没有预安装的 NumPy 可供您使用吗？
该错误可能是由不正确的LD_LIBRARY_PATH设置引起的。您是否从笔记本安装了本地 NumPy，现在正在尝试从 Spark-submit 使用它？这些是不同的执行环境，它们可能不会共享相同的LD_LIBRARY_PATH设置。