Tokenization methods should adapt their merging criteria based on token characteristics—shorter tokens and domain-specific patterns need different frequency thresholds to get fair representation in the vocabulary.
This paper introduces AdaBPE, an improved tokenization method that adapts how subword units are created based on token length and type (general vs. domain-specific language).