The paper "Greed is All You Need: An Evaluation of Tokenizer Inference Methods" by Omri Uzan, Craig W. Schmidt, Chris Tanner, and Yuval Pinter explores the effectiveness of different inference methods for subword tokenizers such as BPE and WordPiece. The authors conduct a controlled analysis of seven tokenizer inference methods across four algorithms and three vocabulary sizes, using a novel intrinsic evaluation suite tailored for English. The evaluation combines measures from morphology, cognition, and information theory. Key findings include: - Greedy inference methods perform surprisingly well, outperforming other methods in morphological alignment. - SaGe, a contextually-informed tokenizer, significantly outperforms other methods in morphological alignment. - Inference methods that minimize token count perform well on cognitive metrics, aligning with human preferences for minimal word segmentation. - The choice of inference method should be aligned with the task and vocabulary to optimize performance. The study highlights the importance of selecting an appropriate inference method for subword tokenizers, particularly for tasks that require morphological alignment. The authors also provide a benchmark for future research and applications, emphasizing the need for a diverse array of languages in future evaluations.The paper "Greed is All You Need: An Evaluation of Tokenizer Inference Methods" by Omri Uzan, Craig W. Schmidt, Chris Tanner, and Yuval Pinter explores the effectiveness of different inference methods for subword tokenizers such as BPE and WordPiece. The authors conduct a controlled analysis of seven tokenizer inference methods across four algorithms and three vocabulary sizes, using a novel intrinsic evaluation suite tailored for English. The evaluation combines measures from morphology, cognition, and information theory. Key findings include: - Greedy inference methods perform surprisingly well, outperforming other methods in morphological alignment. - SaGe, a contextually-informed tokenizer, significantly outperforms other methods in morphological alignment. - Inference methods that minimize token count perform well on cognitive metrics, aligning with human preferences for minimal word segmentation. - The choice of inference method should be aligned with the task and vocabulary to optimize performance. The study highlights the importance of selecting an appropriate inference method for subword tokenizers, particularly for tasks that require morphological alignment. The authors also provide a benchmark for future research and applications, emphasizing the need for a diverse array of languages in future evaluations.