I Built an Agent Eval Harness. Real Agents Broke the Clean Version of the Story

A developer released AgentEval Forge, an open-source harness for evaluating AI agents, on PyPI and as a public repo.

It includes scenario packs, scoring, regression tests, adversarial testing, security controls, CI integration, and adapters for multiple agent frameworks.

Field tests against real LangGraph and PydanticAI repositories exposed messy integration challenges, shifting the project from a scoring tool into a broader reality check.

The takeaway: agent evaluation has to test the full execution run, not just final answers, because real integrations reveal reliability, security, and tooling gaps that synthetic tests can miss.