Tag: tokenizer

A step-by-step guide to building a Thai multilingual subword tokenizer based on a BPE algorithm trained on Thai and English ...

Introduction Large Language Models are known for their text-generation capabilities. They are trained with millions of tokens during the pre-training ...