War Story: How a Kubernetes 1.32 Node OOM Kill Cascaded Into a 2-Hour Outage for Our Video Streaming Service

TL;DR AI
2 min readKey summary
A Kubernetes 1.32 kubelet memory-accounting bug in cgroups v2 triggered a node OOM kill at a major video streaming service.
The incident cascaded across 18 availability zones, cutting traffic by 92% and affecting up to 4.2 million concurrent viewers.
Root cause was underreported memory usage in sidecar-heavy pods, especially with istio-proxy and linkerd-proxy containers.
Adding pod-level memory limits with extra headroom significantly reduced failures and improved resilience.
