Tensorflow unicode文本编码解码

Question

我刚开始处理西里尔文。文本预处理后无法正确打印俄语文本。如何在文本加载期间设置编码?

import pathlib
text = pathlib.Path('rus.txt').read_text(encoding='utf-8')
lines = text.splitlines()
pairs = [line.split('t') for line in lines]
inp = [inp for targ, inp, tag in pairs]
targ = [targ for targ, inp, tag in pairs]
inp[:20]

Output1:

['Марш!',  'Иди.',  'Идите.',  'Здравствуйте.',  'Привет!',  'Хай.', 
'Здрасте.',  'Здоро́во!',  'Приветик!',  'Беги!',  'Бегите!',...

创建数据集:

BUFFER_SIZE = len (inp)
BATCH_SIZE = 64

dataset = tf.data.Dataset.from_tensor_slices((inp, targ)).shuffle(BUFFER_SIZE)
dataset = dataset.batch(BATCH_SIZE)
for example_input_batch, example_target_batch in dataset.take(1):
print(example_input_batch[:5]) --Russian input
print()
print(example_target_batch[:5]) --English target
break

Output2:

tf.Tensor(
[b'xd0xa2xd0xbexd0xbc xd0xbfxd0xbexd1x81xd1x82xd1x83xd0xbfxd0xb8xd0xbb xd1x85xd0xbexd1x80xd0xbexd1x88xd0xbe.'
b'xd0xa2xd1x8b xd1x81xd0xb4xd0xb5xd0xbbxd0xb0xd0xbbxd0xb0 xd1x8dxd1x82xd0xbe xd1x81xd0xbfxd0xb5xd1x86xd0xb8xd0xb0xd0xbbxd1x8cxd0xbdxd0xbe.'
b'xd0xa2xd0xbexd0xbc xd0xb5xd1x89xd1x91 xd0xbdxd0xb5 xd0xbcxd0xbexd0xb6xd0xb5xd1x82 xd1x85xd0xbexd0xb4xd0xb8xd1x82xd1x8c xd1x81xd0xb0xd0xbc.'
b'xd0x94xd1x83xd0xbcxd0xb0xd1x8e, xd0xbfxd0xbexd1x80xd0xb0 xd0xbcxd0xbdxd0xb5 xd0xbfxd0xbexd0xb3xd0xbexd0xb2xd0xbexd1x80xd0xb8xd1x82xd1x8c xd0xbexd0xb1 xd1x8dxd1x82xd0xbexd0xb9 xd0xbfxd1x80xd0xbexd0xb1xd0xbbxd0xb5xd0xbcxd0xb5 xd1x81 xd0xbdxd0xb0xd1x87xd0xb0xd0xbbxd1x8cxd0xbdxd0xb8xd0xbaxd0xbexd0xbc.'
b'xd0xaf xd0xbcxd0xbexd0xb3xd1x83 xd1x8dxd1x82xd0xbe xd1x83xd0xbbxd0xb0xd0xb4xd0xb8xd1x82xd1x8c.'], shape=(5,), dtype=string)
tf.Tensor(
[b'Tom did a good thing.' b'You did that on purpose.'
b"Tom can't walk on his own yet."
b"I think it's time for me to talk to the boss about this problem."
b'I can arrange that.'], shape=(5,), dtype=string)

你能告诉我打印俄文有什么问题吗?英文文本打印正常

Answer 1

看起来像无意义的字符串实际上是UTF-8编码的。

例如张量

中的第一个字符串

xd0xa2xd0xbexd0xbc xd0xbfxd0xbexd1x81xd1x82xd1x83xd0xbfxd0xb8xd0xbb xd1x85xd0xbexd1x80xd0xbexd1x88xd0xbe.

是垃圾

Том поÑÑÑпил ÑоÑо

可以正确解码，像这样:

s = 'xd0xa2xd0xbexd0xbc xd0xbfxd0xbexd1x81xd1x82xd1x83xd0xbfxd0xb8xd0xbb xd1x85xd0xbexd1x80xd0xbexd1x88xd0xbe.'
decoded = bytes(s, encoding='latin').decode()
print(decoded)

输出:

Том поступил хорошо.

我不确定如何用Tensorflow做到这一点，但也许tf.strings.unicode_decode可以帮助。

Answer 2

我使用tf.strings.unicode_decode()函数将编码的'xd0xa2xd0xbexd0xbc'类值转换为[1053, 1077, 32, 1076,...类整数张量。我还将结果转换为numpy数组，使其适用于chr()函数，该函数将unicode整数转换为unicode符号。

def decode_string(ints):
strs = [chr(i) for i in ints]
joined = [''.join(strs)]
return joined
decoded = tf.strings.unicode_decode(example_input_batch[:5], 'utf-8').numpy()
decoded_list = [decode_string(ex) for ex in decoded]
print(decoded_list)

结果是:

[['Том был окружён дельфинами.'], ['Бразилия была колонией Португалии.'], ['Скажи Тому, чтобы поторопился.'], ['Я слишком многого прошу?'],...