Verified, then logged.
Sun 22 Nov 2026 · 00:14 KST / 15:14 UTC한국어 | ENSearch /

01 / AI · Analysis

MINE lifts Llama 3.1 8B inference speed 2.52x on RTX 4060 Ti

The researchers' figures come from a 2.1-bit EntQuant checkpoint of Llama 3.1 8B Instruct, and the speedup was 1.48 times on an RTX 5090.

A consumer graphics card with its cooler removed lies on a workbench, showing a close view of the memory chips surrounding its central processor. AI-generated image

The researchers behind MINE reported a 2.52-times speedup in Llama 3.1 8B inference on an RTX 4060 Ti GPU, using a lossless compression method for large language models (LLMs). The speedup was 1.48 times on an RTX 5090, with both results measured against the fastest baseline method (researcher-published figures). The paper appeared on arXiv on 9 Nov 2026, and the authors said MINE is available as open source.

The researchers target local and low-latency inference, where requests cannot be batched. According to the paper, this setting reads every weight, a numerical value the model has learned, from memory for each generated token. That makes memory the bottleneck on speed.

The speedup in the MINE paper varied with the conditions of the run. A writing-evaluation paper posted to arXiv the same day reported results that split by evaluator. Both sets of numbers have to be read with their conditions.

MINE inference speedup by GPU (researcher-reported)

RTX 4060 Ti2.52
RTX 50901.48
Source arxiv.org · as of 2026-11-09 · unit: timesChart: made by LOGIT

GPU compression built on how weights are distributed

The paper describes MINE as "a lossless entropy coding approach". In plain terms, it writes frequent values in shorter form, so data shrinks with nothing lost. Its core, dtANS-LLM, is a GPU scheme adapted from the earlier dtANS for LLM inference.

MINE exploits the fact that weight values stay unevenly distributed after quantisation, the step that stores weights in fewer bits. The design exposes tuning parameters that trade compression ratio against decoding cost.

According to the paper, MINE auto-tunes these parameters for each tensor and GPU to obtain a Pareto front. A Pareto front is the boundary of the best available trade-offs, here between model size and decoding latency. MINE integrates into vLLM, software for serving LLMs, as a plugin and selects the kernel configuration that fits a given memory budget.

Speedups that vary by GPU

The paper said the tests used a 2.1-bit EntQuant checkpoint of Llama 3.1 8B Instruct. Effective bitrate, the number of bits each weight actually takes up, was 2.24 bits on the RTX 4060 Ti. It was 3.43 bits on the RTX 5090 (researcher-published figures).

The researchers wrote that speedups from existing lossless compression are limited to specific data types such as BF16, a floating-point number format, and depend on additional assumptions. The abstract gives the comparison point only as the "fastest baseline". It does not state that method's name or type.

A writing test that split its evaluators

Another paper posted to arXiv the same day tested whether reasoning-enhanced models also write short stories with more psychological depth. It compared DeepSeek-R1 with DeepSeek-V3, and GPT-5 with GPT-4o. Seven human annotators and an ensemble of LLM judges handled 60 head-to-head comparisons with the model names hidden.

In the human ratings, the paper put GPT-5's preference rate against GPT-4o at 60.0-62.9% (researcher-published figures). The range covers two reasoning-effort settings. DeepSeek-R1 trailed DeepSeek-V3 at 42.9% (researcher-published figure).

Krippendorff's alpha, a measure of agreement between raters, was low at 0.070 (researcher-published figure). The authors read this as genuine differences in subjective judgement, not as a failure of the evaluation.

The LLM judges chose the reasoning model in 98.3% of pairs, a far stronger lean than the 55.2% of human votes (researcher-published figures). According to the paper, this bias was consistent across five configurations and partly tracked sentence length.

Numbers that need their conditions attached

The authors of the writing paper wrote that their results suggest reasoning-related quality differences among frontier models are small enough to be "dominated by reader subjectivity". They also wrote that LLM judges can obscure such subjective differences.

The speedups in the MINE abstract likewise come from one model and two GPUs, and the abstract gives no MINE speedup for any other LLM. Speed multiples and preference rates can be read only with their conditions attached.