SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations

Researchers introduced SynAE, a framework for judging synthetic data quality in tool-calling agent benchmarks.

SynAE evaluates validity, fidelity, and diversity across instructions, tool calls, final outputs, and downstream performance.

The paper argues that no single metric is enough to capture synthetic dataset quality.

This matters as teams increasingly use synthetic traces when real production data is unavailable or sensitive.