Stop Flying Blind: We Built an LLM Evaluation Framework That Works Across 17+ Agent Frameworks

Custom Evals is a lightweight open-source framework for evaluating LLM and agent outputs across 17+ frameworks.

It combines four metric layers: deterministic checks, LLM-as-judge, NLP similarity, and OCR/document metrics.

The library is designed to work without a backend, dashboard, or required test runner.

It fills a production gap by giving teams a unified way to test behavior before deployment across different stacks.