Pandas HDFStore:追加失败，当min_itemsize设置为字符串列的最大值时

Question

我正在检测多个数据帧的所有字符串列的最大长度，然后尝试构建HDFStore：

import pandas as pd
# Detect max string length for each column across all DataFrames
max_lens = {}
for df_path in paths:
df = pd.read_pickle(df_path)
for col in df.columns:
ser = df[col]
if ser.dtype == 'object' and isinstance(
ser.loc[ser.first_valid_index()], str
):
max_lens[col] = max(
ser.dropna().map(len).max(), max_lens.setdefault(col, 0)
)
print('Setting min itemsizes:', max_lens)
hdf_path.unlink()  # Delete of file for clean retry
store = pd.HDFStore(hdf_path, complevel=9)
for df_path in paths:
df = pd.read_pickle(df_path)
store.append(hdf_key, df, min_itemsize=max_lens, data_columns=True)
store.close()

检测到的最大字符串长度如下所示：

max_lens = {'hashtags': 139,
'id': 19,
'source': 157,
'text': 233,
'urls': 2352,
'user_mentions_user_ids': 199,
'in_reply_to_screen_name': 17,
'in_reply_to_status_id': 19,
'in_reply_to_user_id': 19,
'media': 286,
'place': 56,
'quoted_status_id': 19,
'user_id': 19}

然而，我仍然收到此错误：

ValueError: Trying to store a string with len [220] in [hashtags] column but
this column has a limit of [194]!
Consider using min_itemsize to preset the sizes on these columns

这很奇怪，因为检测到的最大hashtags长度是 139。

Answer 1

HDF以UTF-8存储字符串，因此您需要将字符串编码为UTF-8，然后找到最大长度。

a_pandas_string_series.str.encode('utf-8').str.len().max()