Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content

Ingo Ziegler, Martin Krebs, Desmond Elliott|July 17, 2026arXiv

Key Takeaway

Token, byte, and pixel encodings each have different strengths depending on your task and capacity constraints—there's no universally best choice, only task-specific tradeoffs.

Summary

This paper compares three ways to encode text for language models—tokens, bytes, and pixels—by controlling both the linguistic content and model capacity. Using parallel sentences across 13 languages, the researchers measure how well each encoding preserves information under compression.

data evaluation multimodal

Key Terms

tokenization rate-utility-tradeoff information-bottleneck cross-lingual-alignment