Verified, then logged.
Sun 22 Nov 2026 · 00:35 KST / 15:35 UTC한국어 | ENSearch /

Breaking01 / AI

Small model decisions remain sensitive to who takes part

Pooling evaluations left participant effects intact, prompting researchers to recommend candidate sets and margins over forced rankings.

AI-generated editorial illustration for this article

Repeated evaluations of small language models did not remove differences in choices caused by participant selection, researchers reported in a paper submitted to arXiv on 12 Nov 2026. Pooling the results left that sensitivity intact.

The researchers evaluated restaurant reviews using small open-weight language models, whose model weights are publicly available. They used 474 TripAdvisor reviews in London. The Madrid experiment used 600 reviews.

According to the paper, resampling reviewers in London kept the same top restaurant in 47–81% of cases. Resampling evaluation runs kept it in 86–100% of cases. Combining model outputs did not outperform the best contributing model.

The researchers reported that accounting for uncertainty from resampling participants left no single winner. In London, all restaurants remained candidates. They recommended reporting candidate sets and the margins between options rather than forcing a ranking.

The study examined collective decision-making, which brings together people's opinions to reach a judgement. The researchers considered running small open-weight models on an institution's own infrastructure. They explained that this approach keeps the model, inference settings and opinions under the institution's control.

The abstract does not report performance on decision-making tasks beyond restaurant reviews.