Breaking01 / AI
KIWI-EP cuts the time vLLM takes to generate an output token
KIWI-EP changes how tokens are sent to expert models and how the results are collected.
Researchers said on 19 Nov 2026 that KIWI-EP cut the time per output token by up to 29% in the vLLM inference engine.
The paper describes KIWI as a framework that lets a graphics processing unit (GPU) hand off communication tasks to a central processing unit (CPU).
The researchers said they redesigned how KIWI-EP sends tokens to expert models and collects the results from each expert.
In benchmarks on AMD GPUs, the researchers reported up to 2.3 times lower expert-parallel communication latency compared with the best existing implementation.
The paper’s abstract did not disclose which models or input lengths the researchers used to measure time per output token.