错误 Azure Synapse 笔记本：导入错误：令牌化器>=0.10.1，<0.11 是此模块正常运行所必需的，但发现分词器==0.9.2

Question

我正在尝试在Azure synapse Notebook中执行以下命令。

from transformers import AutoTokenizer, AutoModelForMaskedLM 
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")

但是，我得到了下面的错误:

ImportError: tokenizers>=0.10.1,<0.11 is required for a normal functioning of this module, but found tokenizers==0.9.2.
Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master
Traceback (most recent call last):
File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/__init__.py", line 43, in <module>
from . import dependency_versions_check
File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/dependency_versions_check.py", line 41, in <module>
require_version_core(deps[pkg])
File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/utils/versions.py", line 120, in require_version_core
return require_version(requirement, hint)
File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/utils/versions.py", line 114, in require_version
_compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/utils/versions.py", line 50, in _compare_versions
f"{requirement} is required for a normal functioning of this module, but found {pkg}=={got_ver}.{hint}"
ImportError: tokenizers>=0.10.1,<0.11 is required for a normal functioning of this module, but found tokenizers==0.9.2.
Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master

即使在我升级了转换器和标记器版本并通过下面的代码验证它之后，上面的错误仍然存在:

import pkg_resources
for d in pkg_resources.working_set:
print(d)

以上代码的示例输出:

transformers 4.10.3
tokenizers 0.10.3

有什么想法或建议吗?提前谢谢。

Answer 1

您可以在控制台中尝试此命令:

pip install -I transformers --no-cache-dir --force-reinstall

Answer 2

use

pip list | grep packaging

检查您的包装版本是否高于21.3

尝试重新安装您的包，像这样

pip install packaging==21.3

此问题与新打包版本检查中的格式要求有关

我想我应该添加一个评论，而不是在这里添加一个答案，因为这个答案可能不能解决你的问题，但是我没有足够的声誉，所以很抱歉。

Answer 3

在我的案例中，pip install packaging==21.3成功了。

只是因为软件包版本不一致。

只要确保你有正确版本的依赖，你可以检查pipdeptree如果你没有使用pip install pipdeptree的pipdeptree。