Breaking01 / AI
Apple paper tests LLMs against humans on translation scoring
The paper argues that LLMs need thorough evaluation across language pairs and domains before they replace human evaluation of translations.

Apple's machine learning research site published a paper on 12 Oct 2026 that compared large language models (LLMs) with humans on two schemes for evaluating machine translation.
The schemes are Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA), according to the paper's abstract. Model performance was measured by agreement with humans.
The abstract says LLMs are considered a more efficient and cheaper alternative to human judgment in translation evaluation. That evaluation varies by language pair, domain and level of annotation detail. The authors write that LLMs must be thoroughly evaluated across those dimensions before they can reliably replace human evaluation.
The paper's title states that humans and models are both challenged by translation quality annotation.
The opening of the abstract does not give agreement figures or name the models tested.