Codifying the Judge: Scalable Evaluation via Program Distillation

PAJAMA replaces expensive LLM judges with an executable committee of distilled programs that preserve the original judging logic.

It aggregates program outputs into a verdict and escalates only uncertain cases to an LLM, improving accuracy and efficiency.

Across datasets and model families, it matched a 13B LLM judge while improving routing, throughput, and reward-model training efficiency.

The result is a cheaper, faster, and more transparent alternative for automated evaluation and downstream reward modeling.