In-Place Tokenizer Expansion for Pre-trained LLMs

Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak et al.|July 16, 2026arXiv

Key Takeaway

You can expand a pre-trained model's tokenizer in-place by extending BPE merges and initializing new embeddings as averages of sub-tokens, then fine-tuning—this dramatically reduces token overhead for underrepresented languages and speeds up inference for multilingual users.

Summary

This paper presents a method to expand a language model's tokenizer after pre-training to better support additional languages. By continuing BPE merges on a multilingual corpus and carefully initializing new token embeddings, the authors upgrade an 8B-parameter model from a smaller to 128K-token vocabulary.

efficiency training

Key Terms

tokenization bpe-merges vocabulary-expansion embedding-initialization continued-pre-training