The inference method is crucial for NLP models in subword tokenization. Methods such as BPE, WordPiece, and UnigramLM offer different mappings, but their performance differences need to be better understood. Implementations like Huggingface Tokenizers often need to be clearer or limit inference options, which complicates support for vocabulary learning algorithms. It is not known whether a matching inference method is necessary or optimal for tokenizers' vocabularies.
Previous research focused on the development of vocabulary building algorithms such as BPE, WordPieza and UnigramLM, exploring optimal vocabulary size and multilingual vocabularies. Some studies examined the effects of vocabularies on subsequent performance, information theory, and cognitive plausibility. Limited work on inference methods investigated random effects in BPE fusions and sophisticated search algorithms. A comprehensive study comparing inference methods across various vocabularies and sizes should be included.
Researchers at Ben-Gurion University of the Negev Beer Sheva and the Massachusetts Institute of technology conducted a controlled experiment that evaluated seven tokenizer inference methods across four algorithms and three vocabulary sizes. The experiment presents an intrinsic assessment set that combines measures of morphology, cognition, and information theory for English. They have shown that for the most widely used tokenizers, greedy inference works surprisingly well, while SaGe, a contextually informed tokenizer, outperforms others in morphological alignment.
In greedy inference, they only consider and produce one token in each step and define three greedy approaches: First, the “longest prefix” method resembles the approach of selecting the longest token from the vocabulary that is a prefix of the word and iteratively segment the remaining text. Similarly, “longest suffix” specifies the longest word suffix token and continues segmentation iteratively. Finally, the “longest token” selects the longest token contained in the word, adds it to the segmenter, and continues segmenting the remaining characters. These strategies reflect the concept of greedy algorithms that make locally optimal decisions at each step without considering the overall global solution.
The study's extensive evaluation of inference methods on the BPE, UnigramLM, WordPieza, and SaGe vocabularies has revealed variations in performance metrics. Fusion rule-based inference methods often outperform default strategies, particularly notable in morphological alignment. Probability-based methods sometimes assign high probability values to frequently used tokens, which affects the quality of segmentation. SaGe demonstrates superior alignment with morphology. BPE and WordPieza excel in compression but fall behind in cognitive benchmarks. The probability and information-based vocabularies show consistent trends within their respective categories, highlighting the strength of the benchmark.
In conclusion, researchers from Ben-Gurion University of the Negev Beer Sheva and the Massachusetts Institute of technology not only introduced an aggregated benchmark to intrinsically evaluate subword tokenizers, but also emphasized the practical importance of their findings. Selecting inference methods suitable for specific vocabularies and tasks is crucial, and their computational efficiency can help the training of language models by refining tokenization schemes and selecting inference methods. Greedy inference emerges as a favorable option, particularly for morphologically driven tasks, even for tokenizers trained on different targets.
Review the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on Twitter and Google news. Join our 38k+ ML SubReddit, 41k+ Facebook community, Discord Channeland LinkedIn Grabove.
If you like our work, you will love our Newsletter..
Don't forget to join our Telegram channel
You may also like our FREE ai Courses….
Asjad is an internal consultant at Marktechpost. He is pursuing B.tech in Mechanical Engineering at Indian Institute of technology, Kharagpur. Asjad is a machine learning and deep learning enthusiast who is always researching applications of machine learning in healthcare.
<!– ai CONTENT END 2 –>
