02 / IT · Analysis
Moreh unveils AI inference tech with up to 1.66× throughput
A small auxiliary model predicts several tokens ahead, and a large model checks them together.

Moreh, a Korean AI software company, presented research at the SC26 supercomputing conference that reported up to 1.66 times the inference throughput for large language models, according to TechM.
The approach reduces the computation needed when a large language model generates text in sequence, TechM reported.
TechM identified the subject of the research as TIDE, a self-improving inference framework that uses speculative decoding.
Token prediction by the auxiliary model
In TechM’s description of TIDE, a small auxiliary model predicts several tokens ahead. Tokens are units of text that a model processes.
The large model then checks those predictions together, according to TechM.
The basis for the peak throughput figure
TechM reported that Moreh used data generated during inference to increase throughput.
TechM’s report did not specify the baseline for the peak throughput figure.