Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization

A TinyStories study finds the main quality bottleneck in compressed short-text generation is the codec, not the latent generator.

Using a staged diagnosis, researchers separated codec reconstruction quality from latent generation quality in a 64-to-16 pipeline.

Codec reconstruction caused the largest drop in external GPT-2-based text quality, while code-space diffusion beat token-space diffusion.

Geometry-focused regularization improved latent-side proxies but did not improve the final decoded text.