01 / AI · Analysis
Translation compression quality depends on the metric
A translation model compression study found disagreement between automatic metrics and used candidate reranking to improve both compressed systems.

Researchers reported results for 8-bit and 4-bit translation model compression under translation competition conditions.
The study's abstract states that compression lowers serving costs, but a single automatic metric may miss quality loss. Its results make the choice of quality measure part of the assessment of compression.
The researchers reported disagreement between metrics and assigned separate measures to primary evaluation and cross-checking. They also assessed whether selecting again from candidate translations improved the compressed systems.
Metrics give different judgements on compression
The researchers reported that paired bootstrap analysis, which repeatedly resamples paired results for comparison, exposed disagreement between metrics over 4-bit quantisation. Quantisation reduces the numerical precision used to represent model values.
The researchers selected MetricX24-XXL as their primary metric, guided by WMT24 findings that compared metrics with human assessments. They used CometKiwi-XXL as a cross-check. The disagreement means that a quality judgement depends on the measure used as well as the compression setting.
Reranking improves compressed translation systems
The abstract identifies google/gemma-3-12b-it as the model being compressed, with an original size of 24.4 GB in the bf16 numerical storage format. The researchers removed its unused vision component and vocabulary outside the translation task.
The abstract reports no distinguishable quality loss with int8, which uses lower-precision weights. It reports a small remaining difference with the lower-precision int4 setting. These are researcher-reported evaluation findings.
The researchers reported that Cometoid reranking—selecting again from candidate translations—improved evaluation results for both compressed configurations on the WMT25 test set. The abstract does not disclose scores for individual translation directions.
According to the abstract, both configurations remained competitive with bf16 in post-submission WMT26 evaluation using metrics that do not require reference translations. The researchers treated the small difference between the reranked configurations as noise. The relevant comparison in this study therefore combines the storage format with the procedure used to select the translation.