Google announces "multi-token prediction," a technology that uses a small AI to generate drafts and speed up large AI

Google introduced Multi-token Prediction for Gemma 4, a speculative decoding approach that lets a small drafter propose several tokens ahead.

The main model verifies those tokens in parallel, reducing inference latency while preserving output quality.

Google says the method delivered up to about 3x speedups in tests on some hardware, including NVIDIA RTX PRO 6000, Pixel TPU, and NVIDIA A100.

Draft models for several Gemma 4 variants are now available on Hugging Face.