CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

Researchers proposed CoRT, a token-level credit assignment method for rubric-guided GRPO in language models.

It compares the same sampled response under the original rubric prompt and a criteria-free prompt to compute token-wise likelihood contrasts.

Those contrasts are turned into bounded weights that spread a response-level advantage across individual tokens.

In experiments, CoRT beat matched response-level GRPO in most settings and was competitive with learned token-level baselines.

The approach may make RL training for instruction-tuned models more precise without needing a separate token scorer.