Real-time LLM Inference on Standard GPUs: 3k tokens/s per request | Hacker News

A startup claimed real-time LLM inference at about 3,000 tokens per second on standard GPUs.

Hacker News commenters questioned the benchmark setup, model sizes, and hardware behind the demo.

The company said it was only a tech preview and argued the approach could scale to much larger frontier MoE models.

The debate highlights the promise of cheaper, low-latency model serving on commodity GPUs, alongside concerns about fairness and scalability.