TFX评估器似乎不识别来自ResolverNode的基线模型输出

Question

我想在TFX中使用模型评估器组件的验证功能(模型差异或模型比较)，因此我使用了TFX中出租车模板的基本代码。

问题是，当Evaluator组件在GCP上的Kubeflow中运行时，会在日志中抛出下一个错误消息:

ERROR:absl:There are change thresholds, but the baseline is missing. This is allowed only when rubber stamping (first run).
WARNING:absl:"maybe_add_baseline" and "maybe_remove_baseline" are deprecated,
please use "has_baseline" instead.
INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
name: "candidate"
signature_name: "my_model_validation_signature"
label_key: "n_trips"
}
slicing_specs {
}
metrics_specs {
metrics {
class_name: "MeanSquaredError"
threshold {
value_threshold {
upper_bound {
value: 10000.0
}
}
}
}
}
INFO:absl:ModelSpec name "candidate" is being ignored and replaced by "" because a single ModelSpec is being used

查看TFX repo 1中Evaluator组件执行器的源代码第138行:

has_baseline = bool(input_dict.get(BASELINE_MODEL_KEY))

然后在第141行输入一个函数:

eval_config = tfma.update_eval_config_with_defaults(eval_config, has_baseline=has_baseline)

然后抛出引用的错误消息，只有在满足下一个条件时才有可能，从TFX repo 2

if (not has_baseline and has_change_threshold(eval_config) and
not rubber_stamp):
# TODO(b/173657964): Raise an error instead of logging an error.
logging.error('There are change thresholds, but the baseline is missing. '
'This is allowed only when rubber stamping (first run).')

事实上，这是我在日志中得到的错误，模型被评估了，但没有与基线进行比较，即使我按照示例代码所指示的方式提供它，例如:

eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec(name=tfma.CANDIDATE_KEY,label_key='n_trips', 
signature_name='my_model_validation_signature'),
tfma.ModelSpec(name=tfma.BASELINE_KEY, label_key='n_trips', 
signature_name='my_model_validation_signature', is_baseline=True)
],
slicing_specs=[tfma.SlicingSpec()],
metrics_specs=[
tfma.MetricsSpec(metrics=[
tfma.MetricConfig(
class_name="MeanSquaredError",#'mean_absolute_error',
threshold=tfma.MetricThreshold(
value_threshold=tfma.GenericValueThreshold(
upper_bound={'value': 10000}),

change_threshold=tfma.GenericChangeThreshold(
direction=tfma.MetricDirection.LOWER_IS_BETTER,
relative={'value':1})
)
)
])
])

evaluator = Evaluator(
examples=example_gen.outputs['examples'],
model=trainer.outputs['model'],
baseline_model=model_resolver.outputs['model'],
# Change threshold will be ignored if there is no baseline (first run).
eval_config=eval_config)

# TODO(step 6): Uncomment here to add Evaluator to the pipeline.
components.append(evaluator)

它继续…

Answer 1

从0.26.0版本升级到0.27.0版本已解决

问题出现是因为Google Cloud Platform中Kubeflow pipeline的默认笔记本安装了0.26.0版本…