Towards Direct Evaluation of Harness Optimizers via Priority Ranking

Researchers propose priority ranking of harness components as a direct way to measure optimizer skill.

They validate the approach with Shor, a dataset of 182 human-verified scenarios.

Results show ranking performance closely tracks real multi-step optimization gains.

The method provides a cheaper, more diagnostic metric than relying only on final end-to-end results.