Tensorflow预处理将字符串拆分为字符

Question

我想使用TextVectorization预处理层编写，但将字符串拆分为字符。

data = tf.constant(
[
"The Brain is wider than the Sky",
"For put them side by side",
"The one the other will contain",
"With ease and You beside",
]
)
# Instantiate TextVectorization with "int" output_mode
text_vectorizer = preprocessing.TextVectorization(output_mode="int")
# Index the vocabulary via `adapt()`
text_vectorizer.adapt(data)

TextVectorization类有split参数，可以是一个函数。在纯python中，我想这样写:

text_vectorizer = preprocessing.TextVectorization(output_mode="int",split=lambda x:list(x)))

但是我应该如何在TensorFlow的世界里写它呢?

Answer 1

尝试使用tf.strings.regex_replace并首先将每个序列转换为单个字符串，然后再次应用tf.strings.regex_replace将字符串拆分为字符。接下来，使用tf.strings.strip从每个字符串中删除前导和尾随空格。最后，分割并返回字符串:

import tensorflow as tf
def split_chars(input_data):
s = tf.strings.regex_replace(input_data, ' ', '')
tf.print('Single string --> ', s)
s = tf.strings.regex_replace(s, '', ' ')
tf.print('Characters --> ', s)
s = tf.strings.strip(s)
tf.print('Stripped --> ', s)
s = tf.strings.split(s, sep = ' ')
tf.print('Split --> ', s)
return s
data = tf.constant(
[
"The Brain is wider than the Sky",
"For put them side by side",
"The one the other will contain",
"With ease and You beside",
]
)
input_text_processor = tf.keras.layers.TextVectorization(split = split_chars)
input_text_processor.adapt(data)
tf.print(f"Vocabulary --> {input_text_processor.get_vocabulary()}")

Single string -->  ["thebrainiswiderthanthesky" "forputthemsidebyside" "theonetheotherwillcontain" "witheaseandyoubeside"]
Characters -->  [" t h e b r a i n i s w i d e r t h a n t h e s k y " " f o r p u t t h e m s i d e b y s i d e " " t h e o n e t h e o t h e r w i l l c o n t a i n " " w i t h e a s e a n d y o u b e s i d e "]
Stripped -->  ["t h e b r a i n i s w i d e r t h a n t h e s k y" "f o r p u t t h e m s i d e b y s i d e" "t h e o n e t h e o t h e r w i l l c o n t a i n" "w i t h e a s e a n d y o u b e s i d e"]
Split -->  [['t', 'h', 'e', ..., 's', 'k', 'y'], ['f', 'o', 'r', ..., 'i', 'd', 'e'], ['t', 'h', 'e', ..., 'a', 'i', 'n'], ['w', 'i', 't', ..., 'i', 'd', 'e']]
Vocabulary --> ['', '[UNK]', 'e', 't', 'i', 'h', 's', 'n', 'o', 'd', 'a', 'r', 'y', 'w', 'b', 'u', 'l', 'p', 'm', 'k', 'f', 'c']