"ner_ontonotes_bert_mult"模型定制列车

Question

当我想用我的自定义数据集训练这个'ner_ontonotes_bert_mult'模型时，它显示了下面的错误。(我已经将数据集保存在[deeppavlov文档][1]中提到的~.deeppavlovdownloadsontonotes文件夹中。)

PS C:UserssghantaDesktopNER> & c:/Users/sghanta/Desktop/NER/env/Scripts/Activate.ps1
(env) PS C:UserssghantaDesktopNER> & c:/Users/sghanta/Desktop/NER/env/Scripts/python.exe c:/Users/sghanta/Desktop/NER/train_model.py
C:UserssghantaDesktopNERenvlibsite-packagesnumpy_distributor_init.py:32: UserWarning: loaded more than 1 DLL from .libs:
C:UserssghantaDesktopNERenvlibsite-packagesnumpy.libslibopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll
C:UserssghantaDesktopNERenvlibsite-packagesnumpy.libslibopenblas.WCDJNK7YVMPZQ2ME2ZZHJJRJ3JIKNDB7.gfortran-win_amd64.dll
stacklevel=1)
Traceback (most recent call last):
File "c:/Users/sghanta/Desktop/NER/train_model.py", line 12, in <module>
ner_model = train_model(configs.ner.ner_ontonotes_bert_mult)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlov__init__.py", line 29, in train_model
train_evaluate_model_from_config(config, download=download, recursive=recursive)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcorecommandstrain.py", line 92, in train_evaluate_model_from_config
data = read_data_by_config(config)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcorecommandstrain.py", line 58, in read_data_by_config
return reader.read(data_path, **reader_config)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovdataset_readersconll2003_reader.py", line 56, in read
dataset[name] = self.parse_ner_file(file_name)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovdataset_readersconll2003_reader.py", line 106, in parse_ner_file
raise Exception(f"Input is not valid {line}")
Exception: Input is not valid 
O
(env) PS C:UserssghantaDesktopNER> 

清理数据集后，上面的错误已经消失，但这是新的错误。新的错误

2021-08-12 02:43:35.335 ERROR in 'deeppavlov.core.common.params'['params'] at line 112: Exception in <class 'deeppavlov.models.bert.bert_sequence_tagger.BertSequenceTagger'>
Traceback (most recent call last):
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonclientsession.py", line 1365, in _do_call
return fn(*args)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonclientsession.py", line 1350, in _run_fn
target_list, run_metadata)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonclientsession.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [13,13] rhs shape= [37,37]
[[{{node save/Assign_76}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythontrainingsaver.py", line 1290, in restore
{self.saver_def.filename_tensor_name: save_path})
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonclientsession.py", line 956, in run
run_metadata_ptr)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonclientsession.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonclientsession.py", line 1359, in _do_run
run_metadata)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonclientsession.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [13,13] rhs shape= [37,37]
[[node save/Assign_76 (defined at C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonframeworkops.py:1748) ]]
Original stack trace for 'save/Assign_76':
File "c:/Users/sghanta/Desktop/NER/train_model.py", line 12, in <module>
ner_model = train_model(configs.ner.ner_ontonotes_bert_mult)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlov__init__.py", line 29, in train_model
train_evaluate_model_from_config(config, download=download, recursive=recursive)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcorecommandstrain.py", line 121, in train_evaluate_model_from_config
trainer.train(iterator)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcoretrainersnn_trainer.py", line 334, in train
self.fit_chainer(iterator)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcoretrainersfit_trainer.py", line 104, in fit_chainer
component = from_params(component_config, mode='train')
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcorecommonparams.py", line 106, in from_params
component = obj(**dict(config_params, **kwargs))
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcoremodelstf_backend.py", line 76, in __call__
obj.__init__(*args, **kwargs)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcoremodelstf_backend.py", line 28, in _wrapped
return func(*args, **kwargs)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovmodelsbertbert_sequence_tagger.py", line 529, in __init__
**kwargs)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovmodelsbertbert_sequence_tagger.py", line 259, in __init__
self.load()
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcoremodelstf_backend.py", line 28, in _wrapped
return func(*args, **kwargs)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovmodelsbertbert_sequence_tagger.py", line 457, in load
return super().load(exclude_scopes=exclude_scopes, **kwargs)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcoremodelstf_model.py", line 251, in load
return super().load(exclude_scopes=exclude_scopes, **kwargs)
File "C:UserssghantaDesktopNERenvlibsite-packagesdeeppavlovcoremodelstf_model.py", line 54, in load
saver = tf.train.Saver(var_list)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythontrainingsaver.py", line 828, in __init__
self.build()
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythontrainingsaver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythontrainingsaver.py", line 878, in _build
build_restore=build_restore)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythontrainingsaver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythontrainingsaver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythontrainingsavingsaveable_object_util.py", line 73, in restore
self.op.get_shape().is_fully_defined())
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonopsstate_ops.py", line 227, in assign
validate_shape=validate_shape)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonopsgen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonframeworkop_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonutildeprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonframeworkops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonframeworkops.py", line 3426, in _create_op_internal
op_def=op_def)
File "C:UserssghantaDesktopNERenvlibsite-packagestensorflow_corepythonframeworkops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()

谁能解释一下怎么解决它?[1]: http://docs.deeppavlov.ai/en/master/features/models/ner.html

Answer 1

conll2003_reader数据集读取器无法解析以下行:

O

conll2003_reader数据集阅读器期望该行为空或包含一个标记和一个标签。在你的情况下，只有标签存在。

所以，我建议用标签清理你的数据中的空行。

deepppavlov文档中的示例文本:

EU B-ORG
rejects O
the O
call O
of O
Germany B-LOC
to O
boycott O
lamb O
from O
Great B-LOC
Britain I-LOC
. O
China B-LOC