What Is Tokenization Drift and How to Fix It?

The article explains tokenization drift: tiny formatting changes in prompts can alter token IDs and model inputs.

GPT-2 tokenizer examples show that space-prefixed words can split into different token sequences than the same words without leading spaces.

This drift can change sequence length and model behavior, making outputs less stable and prompt reliability harder to manage.

The piece also describes measuring drift and using prompt optimization loops to keep inputs consistent across models like GPT-4, LLaMA, and Mistral.