我已经安装了雪花sqlachemy包。并成功开发了将csv文件上传到snowflake的脚本。
但当我将脚本上传到aws-glue并提供轮子文件时，我遇到了KeyError: 'snowflake-connector-python。
知道吗？
这是完整的错误信息：
Traceback (most recent call last):
File "/tmp/runscript.py", line 123, in <module>
runpy.run_path(temp_file_path, run_name='__main__')
File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 250, in <module>
File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 39, in main
File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 233, in load_data
File "/glue/lib/installation/sqlalchemy/engine/__init__.py", line 479, in create_engine
return strategy.create(*args, **kwargs)
File "/glue/lib/installation/sqlalchemy/engine/strategies.py", line 61, in create
entrypoint = u._get_entrypoint()
File "/glue/lib/installation/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
cls = registry.load(name)
File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 222, in load
return self.impls[name]()
File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 245, in load
mod = compat.import_(modulepath)
File "/glue/lib/installation/snowflake/sqlalchemy/__init__.py", line 8, in <module>
from . import snowdialect
File "/glue/lib/installation/snowflake/sqlalchemy/snowdialect.py", line 27, in <module>
from snowflake.connector.constants import UTF8
File "/glue/lib/installation/snowflake/connector/__init__.py", line 17, in <module>
from .connection import SnowflakeConnection
File "/glue/lib/installation/snowflake/connector/connection.py", line 43, in <module>
from .cursor import LOG_MAX_QUERY_LENGTH, SnowflakeCursor
File "/glue/lib/installation/snowflake/connector/cursor.py", line 47, in <module>
from .arrow_result import ArrowResult
File "src/snowflake/connector/arrow_result.pyx", line 16, in init snowflake.connector.arrow_result
File "/glue/lib/installation/snowflake/connector/options.py", line 36, in <module>
_pandas_extras = pkg_resources.working_set.by_key['snowflake-connector-python']._dep_map['pandas']
KeyError: 'snowflake-connector-python'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 142, in <module>
raise e_type(e_value).with_traceback(new_stack)
File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 250, in <module>
File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 39, in main
File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 233, in load_data
File "/glue/lib/installation/sqlalchemy/engine/__init__.py", line 479, in create_engine
return strategy.create(*args, **kwargs)
File "/glue/lib/installation/sqlalchemy/engine/strategies.py", line 61, in create
entrypoint = u._get_entrypoint()
File "/glue/lib/installation/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
cls = registry.load(name)
File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 222, in load
return self.impls[name]()
File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 245, in load
mod = compat.import_(modulepath)
File "/glue/lib/installation/snowflake/sqlalchemy/__init__.py", line 8, in <module>
from . import snowdialect
File "/glue/lib/installation/snowflake/sqlalchemy/snowdialect.py", line 27, in <module>
from snowflake.connector.constants import UTF8
File "/glue/lib/installation/snowflake/connector/__init__.py", line 17, in <module>
from .connection import SnowflakeConnection
File "/glue/lib/installation/snowflake/connector/connection.py", line 43, in <module>
from .cursor import LOG_MAX_QUERY_LENGTH, SnowflakeCursor
File "/glue/lib/installation/snowflake/connector/cursor.py", line 47, in <module>
from .arrow_result import ArrowResult
File "src/snowflake/connector/arrow_result.pyx", line 16, in init snowflake.connector.arrow_result
File "/glue/lib/installation/snowflake/connector/options.py", line 36, in <module>
_pandas_extras = pkg_resources.working_set.by_key['snowflake-connector-python']._dep_map['pandas']
KeyError: KeyError('snowflake-connector-python',)
这是我的要求.txt
snowflake-connector-python
snowflake-sqlalchemy
sqlalchemy
pyarrow
pandas
asn1crypto==1.4.0
awscli==1.18.140
awswrangler==1.9.3
azure-common==1.1.25
azure-core==1.8.1
azure-storage-blob==12.5.0
boto3==1.14.63
botocore==1.17.63
certifi==2020.6.20
cffi==1.14.2
chardet==3.0.4
colorama==0.4.3 ; python_version != '3.4'
cryptography==2.9.2
docutils==0.15.2
fsspec==0.8.2
idna==2.9
isodate==0.6.0
jmespath==0.10.0
msrest==0.6.19
numpy==1.19.2
oauthlib==3.1.0
oscrypto==1.2.1
packaging==20.4 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
psycopg2-binary==2.8.6
pyasn1==0.4.8
pycparser==2.20
pycryptodomex==3.9.8
pyjwt==1.7.1
pymysql==0.9.0
pyopenssl==19.1.0
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
pyyaml==5.3.1 ; python_version != '3.4'
requests-oauthlib==1.3.0
requests==2.23.0
rsa==4.5 ; python_version != '3.4'
s3fs==0.4.2
s3transfer==0.3.3
six==1.15.0
sqlalchemy-redshift==0.8.1
urllib3==1.25.10
xlrd==1.2.0
从日志文件中，以下是安装的程序包
Successfully installed asn1crypto-1.4.0 awscli-1.18.140 awswrangler-1.9.3 azure-common-1.1.25 azure-core-1.8.1 azure-storage-blob-12.5.0 boto3-1.14.63 botocore-1.17.63 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 cryptography-2.9.2 dependencies-0.1.0 docutils-0.15.2 fsspec-0.8.2 idna-2.9 isodate-0.6.0 jmespath-0.10.0 msrest-0.6.19 numpy-1.19.2 oauthlib-3.1.0 oscrypto-1.2.1 packaging-20.4 pandas-1.1.2 psycopg2-binary-2.8.6 pyarrow-1.0.1 pyasn1-0.4.8 pycparser-2.20 pycryptodomex-3.9.8 pyjwt-1.7.1 pymysql-0.9.0 pyopenssl-19.1.0 pyparsing-2.4.7 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.23.0 requests-oauthlib-1.3.0 rsa-4.5 s3fs-0.4.2 s3transfer-0.3.3 six-1.15.0 snowflake-connector-python-2.3.2 snowflake-sqlalchemy-1.2.3 sqlalchemy-1.3.19 sqlalchemy-redshift-0.8.1 urllib3-1.25.10 xlrd-1.2.0
我建议你做一个"干净"；所有主要库的构建，其中包括pyarrow库。
例如
pip install --upgrade --force-reinstall pandas
pip install --upgrade --force-reinstall pyarrow
pip install --upgrade --force-reinstall snowflake-connector-python
pip install --upgrade --force-reinstall sqlalchemy
pip install --upgrade --force-reinstall snowflake-sqlalchemy
pyarrow的旧版本或丢失版本到处都有一些问题，我认为即使是安装顺序也可能是个问题。全新安装pip或确保您拥有这些库中最新/最棒的库是一个不错的选择。