MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

Researchers defined compositional bias in MLLM-as-a-Judge systems and released MM-JudgeBias to measure it.

The benchmark uses controlled perturbations across queries, images, and responses, including missing, mismatched, and irrelevant cues.

Testing 26 multimodal models showed that many judges exhibit modality neglect and other systematic reliability issues.

MM-JudgeBias provides a common yardstick for evaluating and reducing bias in multimodal AI judges.