Review Arcade: On the Human Alignment and Gameability of LLM Reviews
TL;DR AI
2 min readKey summary
A study of 1,000 ACL 2025 submissions found LLM-generated reviews only weakly matched human judgments.
Review scores varied significantly across prompts and repeated runs, showing poor stability.
Researchers also showed scores could be improved iteratively without substantive paper changes.
The results raise concerns about using LLMs as reliable peer reviewers in conference systems.