Pydantic指定异常以将它们路由到日志

Question

我有一个关于Pydantic的问题，我现在迷路了。我的愿望是创建一个有效的Pydantic对象列表。这是基于是否所有的信息被验证是正确的。如果在验证中有一个错误，我希望将它路由到python的日志模块，这样我就会得到一个日志消息，说明在验证中有一个错误，并且特定的对象被跳过。

这是我创建的当前代码:

from pydantic import BaseModel, Field, validator, ValidationError
import re

class Employee(BaseModel):
Employee_id: str = Field(alias='Employee number')
Username: str
Surname: str
Firstname: str = Field(alias='First name')
Role: str
@validator('Employee_id', pre=True)
def valid_employeeid(cls, employee_id_value):
pattern = re.compile("dddddd")
if not pattern.match(employee_id_value):
if employee_id_value == "":
raise ValueError("Empty mployee ID")
else:
raise ValueError("Not a valid employee ID")
return employee_id_value

employees = [
{
"Employee number": "118791",
"Username": "john.wick",
"Surname": "Wick",
"First name": "John",
"Role": "Shooter"
},
{
"Employee number": "118792",
"Username": "james.bond",
"Surname": "Bond",
"First name": "James",
"Role": "Agent"
},
{
"Employee number": "118g93",
"Username": "jack.reacher",
"Surname": "Reacher",
"First name": "Jack",
"Role": "Shooter"
}
]

pydan_obj = {}
for employee_id in employees:
try:
Employee_data = Employee(**employee_id)
pydan_obj.update({employee_id["Employee number"]: Employee_data})
except ValidationError as e:
print(e)
print(f"Created {len(pydan_obj)} objects")

在这种情况下，我有3名员工，我想将其添加到pydan_obj字典中，在他们的员工编号下。我只想添加具有有效"员工编号"的员工，因此在本例中应该跳过jack reacher，因为他的员工编号并非全是数字。

当前输出为:

1 validation error for Employee
Employee number
Not a valid employee ID (type=value_error)
Created 2 objects

'pydan_obj'只包含两个雇员，这是预期的，但我想写一个日志消息，'jack reacher'被跳过的原因。

我的问题是:首先:我已经创建了一个日志处理程序，但我不确定在上面的代码中初始化或放置它的位置。

Second:当前except捕获每个验证错误。我想把它分成更多的可能性。例如，如果我为姓氏添加一个验证器，我想给它一个自定义异常，创建一个不同的日志消息。

最后:我想用这种方式创建一个员工列表。这个例子中只有3个人，但在我的情况下，我想扩大到1000-2000名员工。这是正确的方法吗?还是有其他方法来保存这些对象?我需要在整个脚本运行时保持它们，因为我在不同的场合需要这些信息。

最诚挚的问候和感谢你的帮助!

Answer 1

好问题

1. 以下是您如何在一般(使用模块级日志)中记录日志:

import logging
# Instantiate the logger by name
logger = logging.getLogger(__name__)
# Now log something
logger.info("this is a message")

2. 以下是日志记录(带错误处理)在您的解决方案中的工作方式:

关于错误处理的注意事项:当您的验证器函数失败时，它会引发一个ValidationError，这是一个错误集合。您正在尝试匹配ValidationError中的特定错误。我为您提供了一种方法，尽管它有点棘手。

import logging
import re
from pydantic import BaseModel, Field, validator, ValidationError

# Instantiate logger
logger = logging.getLogger(__name__)

# Create custom exceptions
class InvalidEmployeeId(ValueError):
...

class EmptyEmployeeId(ValueError):
...

class Employee(BaseModel):
Employee_id: str = Field(alias='Employee number')
Username: str
Surname: str
Firstname: str = Field(alias='First name')
Role: str

@validator('Employee_id')
def valid_employeeid(cls, value):
pattern = re.compile("dddddd")
if not pattern.match(value):
# Raise these specific errors, but you can raise others
if value == "":
raise EmptyEmployeeId("Empty employee ID")
else:
raise InvalidEmployeeId("Not a valid employee ID")
return value
employees = [
...
]

# I renamed some of the variables to make more sense
pydantic_objects = {}
for employee_record in employees:
# Try creating an employee object
try:
employee_obj = Employee.parse_obj(employee_record)
# Assign to your dict
pydantic_objects[employee_obj.Employee_id] = employee_obj
# If validation failed, check for specific errors
# by entering the validation error context, then raising
# "child" errors. (Alternatively, you can compare the error
# context and match the class name.) The reason this is a bit
# complicated is because Pydantic is migrating to a rust backend
# where the validations are performed in rust: https://docs.pydantic.dev/blog/pydantic-v2/#validator-function-improvements
except ValidationError as validation_error:
try:
# Raise the "child" errors from the parent
for e in validation_error.raw_errors:
raise (e.exc)
except InvalidEmployeeId:
logger.warning(
f"Skipping this record (records={employee_record}) "
"EmployeeID is not a valid ID"
)
except EmptyEmployeeId:
logger.warning(
f"Skipping this record (records={employee_record}) "
"EmployeeID is an empty string."
)
# If it fails for a reason you didn't specify, halt the program
except Exception as e:
logger.warning("This is a different warning we didn't catch")
raise Exception(e)

print(f"Created {len(pydantic_objects)} objects")

3. 此解决方案确实适用于许多员工，尽管您可能希望存储"不良记录";在另一个对象中，这样您就可以导出它们，或者进行其他处理。你可能还想报告一些关于数据好坏的指标。