Why MoE Models Get More From Speculative Decoding | Cohere

Cohere’s benchmark shows that speculative decoding speedups in MoE models depend strongly on batch size.

Dense models see steadily smaller gains as batch size rises, while MoE models have a non-monotonic curve with a peak at moderate batch sizes and unusually large gains at batch size 1.

The behavior is attributed to low arithmetic intensity, expert-loading patterns during verification, temporal correlation in routing, and fixed overheads that become nearly free in the bandwidth-bound regime.

In practice, speculative decoding can accelerate MoE inference, but the benefit depends on batch size and routing behavior, which matters for LLM serving optimization.