Verified, then logged.
Sun 22 Nov 2026 · 00:21 KST / 15:21 UTC한국어 | ENSearch /

01 / AI · Analysis

Language model edits depend on evaluation conditions

Papers on refusal removal and vocabulary trimming assessed model behaviour after edits alongside the conditions used to evaluate it.

AI-generated editorial illustration for this article

Researchers published an arXiv study assessing refusal removal using 972 harmful prompts. They counted refusals with an automated judge, but did not measure variation across repeated runs for some methods.

A separate arXiv paper examined vocabulary trimming: removing tokens, the units models use to process text. It identified problems with keeping only tokens found in a user's own writing.

The papers examined different consequences of model edits: refusal responses and the way text is split into tokens. Both made behaviour after modification part of the assessment.

Refusal removal difficulty varied by model

The refusal study found that removing refusals did not consistently become harder in newer models. Its findings qualified the idea that newer releases necessarily resist removal more strongly.

The researchers compared methods that remove a direction associated with refusal from a model's weights. They also evaluated Arbitrary-Rank Ablation (ARA), an optimisation-based refusal removal method.

Each ARA assessment rested on a single run. Some conclusions depended on how much change in the model's output the evaluation allowed, limiting what those results established about removal difficulty.

Vocabulary trimming needed to preserve merge relationships

The vocabulary paper examined the storage burden of unused tokens. In deployments using Thai and English, that burden fell more heavily on smaller models.

The paper reported that existing trimming methods also removed intermediate tokens used to build other tokens through merging. Removing those intermediate steps changed how the model split new text.

The researchers addressed those dependencies through a method called merge closure, which preserves the tokens needed for merging. They also proposed a certificate that uses the text supplied for trimming to estimate how often new text would be split differently.

Both papers treated behaviour after removal as a criterion for judging an edit. Their shared implication was that an edit's apparent success needed to be read alongside its evaluation conditions: how refusal was assessed, or whether token splitting was preserved on new text.