We terminated a TPU mid-training and it recovered in seconds: Introduction to elastic training with MaxText

A MaxText JAX training run on Google Cloud TPUs survived a deliberate worker failure on GKE.

Instead of restarting from scratch, the job resumed in place within seconds through elastic training.

MaxText, PathwayS, and Orbax handled recovery, checkpointing, and state restoration after the TPU node went down.

The demo highlights better fault tolerance for large distributed training jobs, reducing downtime and wasted compute.