Breaking01 / AI
Evaluation methods reverse image AI model rankings
Similar overall scores masked disagreement over individual images, while automated model rankings differed from human assessments.

Researchers reported that evaluation methods distorted automated assessments of image-generating AI in an analysis of 6,240 images.
The paper's abstract describes verifiers as tools that assign scores to generated images. The researchers analysed images from Qwen-Image-2512 and FLUX.2-dev, comparing verifiers and how they were used on a balanced evaluation set of 2,280 images.
According to the paper, similar overall scores masked disagreement over individual images. Every combination of verifier and evaluation procedure narrowed the score gap between the two generators found by human assessors.
The researchers argued that an automated evaluation score does not measure the generator alone. The input prompts, verifier, procedure for using it and scoring rules also contribute to the result.
The paper also reported a comparison using a fixed set of prompts that added Z-Image. Every procedure tested ranked Z-Image and FLUX.2-dev in the opposite order to human assessments.
The researchers did not disclose model-by-model scores for the Z-Image comparison in the abstract.