01 / AI · Analysis
TKNP raises long-context AI throughput by up to 1.7 times
Splitting a model across servers while keeping each request's attention on its assigned server is central to TKNP's throughput comparison.

Researchers reported that Decode Token Parallelism (TKNP) delivered up to 1.7 times the completion throughput of decode context parallelism (DCP) for long-context workloads with many concurrent requests. The paper-published benchmark used matching GPU counts.
The paper explains that serving more simultaneous requests with long contexts requires more storage and attention bandwidth. Attention is the computation that refers to stored context information for each request. These demands make the distribution of work across servers central to the comparison.
The proposed design separates where model weights reside from which server handles a request's attention. It spreads attention work across servers without replicating the model, while keeping each request's attention on its assigned server.
Splitting the model while keeping attention local
According to the abstract, the researchers implemented TKNP in vLLM, software for serving language models, and evaluated it on up to 64 H100 GPUs. The evaluation used Llama 3.1 405B and Nemotron-4 340B.
The paper describes a single model divided into blocks of layers across servers. The server holding a block performs that block's projections and multilayer perceptron (MLP) computations for all requests. These operations transform the model's internal representations.
Each server computes attention for its assigned requests using a locally stored KV cache, which holds information needed to process context, according to the paper. Only intermediate computation values for the current token pass between servers.
Removing the need to combine attention results
The paper also reported up to 2.06 times the throughput of data parallelism, which replicates the model. This paper-published benchmark also used matching GPU counts for long-context workloads with many concurrent requests.
The paper explains that DCP splits context across servers and then combines their partial attention results. TKNP completes each request's attention on its assigned server, removing that cross-server combination step.
The abstract presents the throughput gains as maximum multiples and does not disclose throughput for individual workload conditions. The architectural difference provides context for those reported peaks: TKNP distributes the model while keeping each request's attention computation local to its assigned server.