与每件物品相比，在批量使用transformers标记器时，速度是否有显著提高

Question

在批处理中调用tokenizer的速度是否明显快于在批处理的每个项目上调用它？例如

encodings = tokenizer(sentences)
# vs
encodings = [tokenizer(x) for x in sentences]

Answer 1

我最终只是对两者进行了计时，以防对其他感兴趣

%%timeit
for _ in range(10**4): tokenizer("Lorem ipsum dolor sit amet, consectetur adipiscing elit.")
785 ms ± 24.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
tokenizer(["Lorem ipsum dolor sit amet, consectetur adipiscing elit."]*10**4)
266 ms ± 6.52 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)