How a model upgrade silently broke our extraction prompt (and how we caught it)

TL;DR AI
2 min readKey summary
A customer-support ticket summarization prompt worked on GPT-4o, but after moving to GPT-4.1 it silently renamed the urgency field.
Because the JSON still parsed, downstream logic defaulted every ticket to low urgency until a customer complained.
The story highlights how LLM upgrades can cause output drift without obvious errors, breaking structured workflows.
It argues for baseline comparison tests and CI checks to catch prompt regressions before model swaps ship.
