Verified, then logged.
Sun 22 Nov 2026 · 00:21 KST / 15:21 UTC한국어 | ENSearch /

01 / AI · Analysis

Khmer benchmark drops LLM judges but still needs LLM answers

The judge is gone from scoring, but LLMs still write the reference answers and the human check is one native speaker's ratings.

Seen from behind in a Phnom Penh office, a person compares several blurred columns of written answers on a laptop, with a Cambodian temple roof visible through the window. AI-generated image

Researchers released Khmer PFGen-Bench on arXiv on 9 Nov 2026, a benchmark that scores multi-sentence answers in Khmer without a large language model (LLM) judge, using example answers from nine model families.

Benchmarks for open-ended answers often hand scoring to an LLM, the paper said, and that judge's reliability is uncertain in low-resource languages, meaning languages with little data available for training. The issue is whether rules and statistics can rank models where trust in a judge is uncertain.

Removing the judge from scoring moved the burden onto the reference answers, the example answers that set the scoring standard. The researchers examined how to build those answers in a separate part of the paper. LLMs still write them, and the human check is one native speaker's ratings.

Scoring rules replace the judge

The scorer adapts the one used in PFGen-Bench, a Japanese benchmark, to Khmer, the paper said. PFGen-Bench scores responses with keyword rules and n-gram statistics, which count how often runs of consecutive characters or words appear.

In the paper's description, those rules and statistics do the work of reading and judging a response. The reference answers behind the n-gram statistics, however, are generated by LLMs. The scoring standard therefore still depends on LLM output, even with the judge removed.

The paper described the benchmark as "to our knowledge the first judge-free benchmark". The claim to be first is limited to what the authors know. Its scope is also narrow: multi-sentence answers to questions composed directly in Khmer.

Post-trained models reshape the reference answers

This scoring needs reference answers that are both valid and varied, the researchers explained. Their starting point was that LLMs may not produce such answers in Khmer.

Post-trained models are base models that receive further training after pre-training. According to the paper, they can give more valid but less varied reference answers than their base variants. For the same number of reference answers, mixing model families increases variety.

The benchmark's reference answers came from post-trained models in nine families, the paper said. That setup combines the validity of post-training with the variety of mixed families. The abstract does not give the number of reference answers per family.

Rankings checked against a native speaker

According to results the researchers published, this setup produced model rankings that were stable across reference sets. The rankings were also consistent with a native speaker's ratings. The abstract does not give per-model scores or a ranking table.

The researchers presented the findings as guidance for building judge-free benchmarks in other low-resource languages. The human check behind that guidance is the ratings of one native speaker.