The GKE Upgrade That Took Down Our Production Pods for 45 Minutes

A routine automatic GKE Standard node pool upgrade caused a production disruption when nodes were drained and pods were evicted one by one.

Two critical services — session validation and rate limiting — each had only two replicas, so overlapping evictions left them temporarily unavailable.

The result was about 45 minutes of slow or failing API requests until the upgrade moved past the affected nodes.

The incident highlights that automatic Kubernetes upgrades can still impact production without Pod Disruption Budgets and safe rescheduling design.