01 / AI · Analysis
RoCoKV speeds up reasoning in a test against a full KV cache
Researchers designed RoCoKV to vary how much it compresses the KV cache according to the model’s confidence during inference.
Researchers reported that RoCoKV ran at 1.27 times the speed of a full cache on the AIME25 benchmark with DeepSeek-R1-Distill-Llama-8B.
The researchers said the long reasoning sequences that large models generate while working towards an answer account for most of their inference cost.
The paper’s abstract identifies the KV cache as what RoCoKV compresses. The cache stores intermediate values the model reuses as it generates tokens.
Compression varies with model confidence
The researchers wrote that repeated tokens can still receive high attention scores.
The researchers designed RoCoKV to allocate more cache memory when the model is less confident and compress the cache more aggressively when it is more confident. The method also reduces the influence of tokens that closely resemble recent reasoning content.
The researchers said they integrated RoCoKV into vLLM, a tool for running model inference, as a plugin that works with its existing cache layout.
Speed against a full cache
The abstract reports speed figures for DeepSeek-R1-Distill-Llama-8B in vLLM. The baseline, Full-KV, retains the entire cache.
The researchers reported that RoCoKV ran at 1.43 times the speed of Full-KV on a long synthetic benchmark. They used the same baseline to compare throughput, the amount of work processed in a given time.
The researchers reported higher reasoning accuracy than with existing compression methods when cache space was limited. The abstract does not give the size of the accuracy gain.