How VLAs Fail Differently: Black-Box Action Monitoring Reveals Architecture-Specific Failure Signatures

A study of three vision-language-action robot policies shows failure signals depend on the model architecture, so monitoring should not be one-size-fits-all.

Across 450 episodes, direction reversal emerged as a strong universal predictor of failure.

Jerk was useful mainly for discrete-token models like VQ-BeT, while velocity checks were often weak or ineffective, especially for continuous policies.

The researchers introduced SafeContract, a training-free black-box monitoring toolkit with conformal calibration to better detect robot policy failures.