01 / AI · Analysis
CECAR speeds on-device inference through expert selection
CECAR adds predictions of expert reuse to cache-aware routing for on-device inference.

Researchers reported 12.5 tokens per second for CECAR in a researcher-published benchmark on a graphics card with 16 GB of video memory for on-device inference.
The researchers described a memory constraint for mixture-of-experts (MoE) models, which select among specialised modules called experts. Even when only some experts are active, all expert weights must remain in memory. This makes memory capacity a constraint on running these models on devices.
According to the paper’s abstract, CECAR combines expert selection with cache management. Its design predicts the interval before each expert is used again and uses that prediction to guide selection.
From cache presence to reuse prediction
The researchers reported that cache hit rates have limits even under an optimal replacement policy. Their analysis found those limits insufficient for low-latency on-device inference.
The paper distinguishes cache presence, meaning whether an expert is already cached, from reuse distance, meaning the interval before it is used again. The researchers argued that routing based only on cache presence can make expert selection unstable.
According to the abstract, CECAR uses machine learning to predict each expert’s reuse distance. It prioritises the selection and execution of experts with shorter predicted intervals.
SSD-backed inference and comparison conditions
The paper’s abstract identifies Qwen3-30B-A3B as the model used in the experiment. Experts absent from the graphics card’s video memory are fetched from a solid-state drive (SSD).
The researchers reported a speedup of 4.5 times over conventional MoE inference in their published benchmark. They also reported faster token generation than comparison systems that keep the entire model in main memory, or DRAM.
The paper therefore links expert reuse prediction with faster token generation under the tested memory conditions. Its abstract does not disclose the comparison systems’ throughput in tokens per second or numerical comparisons of output quality, leaving the size of the DRAM comparison advantage and the quality trade-off unspecified.