Hardware-Aware, Dynamic Speculative Decoding (DSD) | Cohere

Cohere introduced dynamic speculative decoding, which adjusts draft token counts based on GPU conditions and batch size.

It increases drafting when inference is memory bandwidth-bound and reduces it when compute-bound, improving efficiency.

The method addresses the fact that fixed speculative decoding can help or hurt depending on hardware utilization and batching.

This adaptive approach could improve production LLM serving and reinforcement learning rollout throughput.