Verified, then logged.
Sun 22 Nov 2026 · 00:21 KST / 15:21 UTC한국어 | ENSearch /

01 / AI · Claim vs. Check · Analysis

KLEAR holds knowledge fixed to test Korean LLMs' application

The design aims to separate the causes of wrong answers that accuracy scores conceal, though the abstract names no evaluated models and gives no scores.

At dusk in a Korean university computer lab, an out-of-focus researcher seen from behind compares a short test question with a more complex one displayed side by side on a monitor. AI-generated image

KLEAR, a framework for evaluating Korean large language models (LLMs) posted on arXiv on 9 Nov 2026, holds the required knowledge fixed and raises only the difficulty of applying it. According to the paper's abstract, the design aims to separate wrong answers caused by missing knowledge from those caused by failed application.

The researchers argued that an accuracy score alone cannot show whether a model lacked the knowledge or failed to use what it knew. KLEAR moves that distinction into the design stage of the test questions.

The researchers first examined what existing Korean tests actually measured.

Existing tests that factual recall can solve

The researchers wrote in the abstract that most questions in existing Korean benchmarks can be answered through "factual recall alone". A model can reach the answer simply by remembering a fact.

Questions that demanded more application also differed in the knowledge they required, according to the researchers' analysis. That made it difficult to isolate application ability from the difference in scores between two questions.

Harder questions built on the same knowledge

According to the paper, KLEAR sets out the knowledge each original question requires as an explicit anchor bundle. It uses that bundle to generate questions requiring more complex application. The result is a contrast set with the same knowledge and different application difficulty.

The researchers said they put the generated candidate questions through automated and human validation and revised them repeatedly using evaluation feedback. In experiments, the finished sets largely met the intended constraints. The abstract does not give the share of questions that fell outside those constraints.

Application failures that persist with knowledge provided

The researchers said they evaluated a range of LLMs on the knowledge-matched sets and that raising application difficulty exposed model failures. Those failures persisted "even when the required knowledge is provided", the abstract states.

The abstract does not name the models evaluated or give scores for each model. Which models are weak at application depends on those figures. The question KLEAR asks is why a model got an answer wrong, not how many points it scored.