SpecMD: A Comprehensive Study on Speculative Expert Prefetching

SpecMD introduces a standardized benchmark for MoE expert cache policies across hardware settings.

The study finds that common temporal-locality heuristics like LRU and LFU often miss the true access patterns of MoE models.

A new eviction policy, Least-Stale, reduces collision misses, improves hit rates, and lowers first-token latency.

Better expert caching is key to turning MoE sparsity into real inference speedups while cutting memory and serving costs.