我有一个使用EMR-6.3.1的EMR集群。我正在使用Python3内核。
我有一个非常简单的引导脚本在S3:
#!/bin/bash
sudo python3 -m pip install Cython==0.29.4 boto==2.49.0 boto3==1.18.50 numpy==1.19.5 pandas==1.3.2 pyarrow==5.0.0
这些是引导日志
+ sudo python3 -m pip install Cython==0.29.4 boto==2.49.0 boto3==1.18.50 numpy==1.19.5 pandas==1.3.2 pyarrow==5.0.0
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
WARNING: The scripts cygdb, cython and cythonize are installed in '/usr/local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/usr/local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script plasma_store is installed in '/usr/local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
和
Collecting Cython==0.29.4
Downloading Cython-0.29.4-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
Requirement already satisfied: boto==2.49.0 in /usr/local/lib/python3.7/site-packages (2.49.0)
Collecting boto3==1.18.50
Downloading boto3-1.18.50-py3-none-any.whl (131 kB)
Collecting numpy==1.19.5
Downloading numpy-1.19.5-cp37-cp37m-manylinux2010_x86_64.whl (14.8 MB)
Collecting pandas==1.3.2
Downloading pandas-1.3.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB)
Collecting pyarrow==5.0.0
Downloading pyarrow-5.0.0-cp37-cp37m-manylinux2014_x86_64.whl (23.6 MB)
Collecting s3transfer<0.6.0,>=0.5.0
Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting botocore<1.22.0,>=1.21.50
Downloading botocore-1.21.65-py3-none-any.whl (8.0 MB)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from boto3==1.18.50) (0.10.0)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/site-packages (from pandas==1.3.2) (2021.1)
Collecting python-dateutil>=2.7.3
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<1.27,>=1.25.4
Downloading urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas==1.3.2) (1.13.0)
Installing collected packages: Cython, python-dateutil, urllib3, botocore, s3transfer, boto3, numpy, pandas, pyarrow
Successfully installed Cython-0.29.4 boto3-1.18.50 botocore-1.21.65 numpy-1.19.5 pandas-1.3.2 pyarrow-5.0.0 python-dateutil-2.8.2 s3transfer-0.5.2 urllib3-1.26.13
从笔记本中导入pandas，看到错误的版本- 1.2.3。此外，我看到pyarrow无法导入。
我已经打印了pandas的导入路径，运行的是哪个python版本，还有sys.path.
import os
import pandas
import sys
print(sys.path)
print(pandas.__version__)
print(os.path.abspath(pandas.__file__))
print(os.popen('echo $PYTHONPATH').read())
print(os.popen('which python3').read())
# sys.path.append('/usr/local/lib64/python3.7/site-packages') # if I add this, pyarrow can import
import pyarrow
['/', '/emr/notebook-env/lib/python37.zip', '/emr/notebook-env/lib/python3.7', '/emr/notebook-env/lib/python3.7/lib-dynload', '', '/emr/notebook-env/lib/python3.7/site-packages', '/emr/notebook-env/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg', '/emr/notebook-env/lib/python3.7/site-packages/IPython/extensions', '/home/emr-notebook/.ipython']
1.2.3
/emr/notebook-env/lib/python3.7/site-packages/pandas/__init__.py
/usr/bin/python3
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-2-aea9862499ce> in <module>
9
10 # sys.path.append('/usr/local/lib64/python3.7/site-packages') # if I add this, pyarrow can import
---> 11 import pyarrow
ModuleNotFoundError: No module named 'pyarrow'
我发现我可以导入pyarrow，如果我添加/usr/local/lib64/python3.7/site-packages到sys.path。这似乎是一个改进，但仍然导入了错误的熊猫版本。
我试过:
- SSH到主节点并修改配置。
sudo python3 -m pip install --user ...export PYTHONPATH=/usr/local/lib64/python3.7/site-packages && sudo python3 -m pip install ...sudo pip3 install --upgrade setuptools && sudo python3 -m pip install ...- 使用pyspark内核并运行
sc.install_pypi_package("pandas==1.3.2")
任何帮助都是感激的。谢谢你。
EMR上的引导配置不是集群等待和EMR步骤开始运行之前的最后一步。
在emr集群中，我发现至少这些包被记录为安装在之后的启动配置运行。我遇到了numpy不升级的问题。
Python packages installed post bootstrap
2022-12-10 00:10:28,250main:花了1分3秒451毫秒安装包:
[emr-scripts, emr-s3-select, laws -sagemaker-spark-sdk，Python27-numpy、python27-sagemaker_pyspark、python37-numpy、Python37-sagemaker_pyspark, emr-ddb, hadoop-yarn-nodemanager, dockerHadoop-yarn, spark-yarn-shuffle, bigtop-utils, cloudwatch-sink，Hadoop, Hadoop -lzo, emr-goodies, emrfs, Hadoop -mapreduce, Hadoop -hdfs，R-core, aws-hm-client, emr- kineesis, hadoop-hdfs-datanode，spark-datanucleus]
在您的第一个集群步骤中添加安装
...
Steps: [
{
'Name': 'Install Pandas',
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar":
"s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar",
"Args": [
"s3://bucket/prefix/install_packages.sh"
]
}
},
]
或者如果你想使用command-runner
{
"Name": "Install Pandas",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"bash",
"-c",
" aws s3 cp s3://bucket/prefix/install.sh .; chmod +x install.sh; ./install.sh; rm install_pandas.sh "
]
}
}
#!/bin/bash
set -x
sudo pip3 freeze # for debugging to view the previous package versions
sudo pip3 uninstall numpy -y -v
sudo yum install python3-devel -y
sudo pip3 install boto3==1.26.26 -v
sudo pip3 install numpy==1.21.6 -v
sudo pip3 install pandas==1.3.5 -v
sudo pip3 freeze # for debugging to view the post-install package versions