Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization

Researchers introduced Verus-SpecGym and Verus-SpecBench to evaluate whether LLMs can turn Codeforces-style problem statements into correct Verus specifications.

They built a 581-task benchmark plus an interactive spec-writing environment, then tested frontier and open-source models with executable specs and adversarial edge-case checks.

Gemini 3.1 Pro performed best overall, but common failures included missing assumptions and wrong handling of valid or invalid cases.

The study also found that LLM-as-a-judge often missed specification errors, showing that automated evaluation still has important blind spots.

Overall, the work suggests LLMs are promising for specification autoformalization, but still too brittle for dependable formal verification without stronger safeguards.