Verified, then logged.
Sun 22 Nov 2026 · 00:24 KST / 15:24 UTC한국어 | ENSearch /

Breaking01 / AI

Models vary in reading context in English and Mandarin tests

Researchers reported that language models were more likely than human participants to take remarks literally in dialogue questions.

A monitor displays a reference image of a speckled ceramic mug alongside several similar mugs, with an amber segmentation mask covering only the matching mug. AI-generated image

Researchers reported in PILAPS, an arXiv preprint posted on 20 Nov 2026 that has yet to be peer-reviewed, that models varied in how well they inferred meaning from context on English and Mandarin Chinese questions. The abstract makes no mention of independent replication.

According to the abstract, the researchers wrote some questions and drew others from dialogue to test what remarks meant in context. They used both culturally adapted and independently sourced questions.

The researchers reported that open-source models had particular difficulty interpreting humour. On dialogue questions, language models were more likely than human participants to take remarks literally.

Asking models to consider what the other person might be thinking reduced the share of wrong answers based on literal readings, the researchers reported. Asking models to spell out their reasoning step by step slightly increased that share.

The paper cited earlier studies that found language models performed well at inferring meaning from context in English. The PILAPS evaluation tested whether that performance held across languages and conversational settings.

The abstract does not give the size of either change in the share of wrong answers based on literal readings.