Verified, then logged.
Sun 22 Nov 2026 · 00:22 KST / 15:22 UTC한국어 | ENSearch /

Breaking01 / AI

AI validation filters out harder language-model test items

Researchers said AI validation removed questions whose answers humans supported, raising language models' benchmark accuracy.

A researcher seen from behind compares blurred sentence pairs and item exclusions before and after AI validation on two monitors in a language-model evaluation lab. AI-generated image

AI validation rejected test items that were 18.9 points harder for language models than accepted items, researchers reported in an abstract published on 11 Nov, comparing items whose answers humans supported.

The study examines minimal-pair benchmarks, which ask models to choose the correct sentence from a pair. It covers BLiMP in English and QFrBLiMP in Quebec French.

The researchers said they compared rejected and accepted items across 10 open language models. Rejected items were harder in all 70 model-validator combinations, according to the researchers' published figures.

The researchers reported that the filter removed items that language models answered incorrectly but humans answered correctly. This raised benchmark accuracy more than removing items at random.

On BLiMP, AI validators required to choose between two sentences agreed with human majority decisions more often than human panels did, according to the abstract. Yet they were less effective than a single human judge at identifying items whose designated answer humans rejected.

The researchers did not disclose accuracy figures before and after filtering in the abstract.