Verified, then logged.
Sun 22 Nov 2026 · 00:23 KST / 15:23 UTC한국어 | ENSearch /

Breaking01 / AI

KIWI-EP cuts the time vLLM takes to generate an output token

KIWI-EP changes how tokens are sent to expert models and how the results are collected.

Researchers said on 19 Nov 2026 that KIWI-EP cut the time per output token by up to 29% in the vLLM inference engine.

The paper describes KIWI as a framework that lets a graphics processing unit (GPU) hand off communication tasks to a central processing unit (CPU).

The researchers said they redesigned how KIWI-EP sends tokens to expert models and collects the results from each expert.

In benchmarks on AMD GPUs, the researchers reported up to 2.3 times lower expert-parallel communication latency compared with the best existing implementation.

The paper’s abstract did not disclose which models or input lengths the researchers used to measure time per output token.