Breaking the Architectural Bottlenecks of Large Language Model Inference

The Mechanics of Draft and Verify

Standard text generation in large language models relies on an autoregressive process. During this sequential operation, an input sequence undergoes a forward pass where it is tokenized, processed through model layers, and converted into a probability distribution. The decoding phase then selects a single token to append to the input before repeating the entire cycle. Because this vanilla approach requires a full model run to produce just one token, it introduces significant latency. Speculative decoding addresses this bottleneck by pairing a smaller, rapid draft model with a larger target model. Acting like a fast typist drafting text for an editor to verify, the smaller model guesses future tokens in parallel, allowing the system to output multiple tokens in the time it would normally take to generate one. This setup typically yields speedups of two to four times without degrading the final output quality.

Overcoming the Shared Vocabulary Constraint

Despite its efficiency, traditional speculative decoding has a major limitation: the draft and target models must share the identical vocabulary. This requirement restricts the choice of drafting models to the same family, such as Llama, StarCoder, or DeepSeek, or forces developers to train a custom drafter from scratch. Training a new model demands significant computational resources, specialized expertise, and time, and the resulting drafter cannot be reused for models with different vocabularies. To solve this issue, researchers from the Weizmann Institute of Science, Intel Labs, and d-Matrix developed three new speculative decoding algorithms designed for heterogeneous vocabularies. Published for the ICML 2025 conference, these methods are entirely lossless, meaning they preserve the exact probability distribution of the target model. They function with off-the-shelf models without needing retraining or architectural modifications, delivering speedups of up to 2.8 times on programming, summarization, and long-context tasks.

Scaling Hardware for Advanced Inference

As language models grow, serving them efficiently requires specialized hardware configurations. For instance, running a model like GPT-OSS-120B with EAGLE-3 or Llama-3.3-70B with draft-target speculative decoding is possible on a single NVIDIA DGX Spark, which contains 128 gigabytes of unified memory shared between the processor and the graphics unit. However, massive models like Qwen3-235B-A22B exceed these limits. Even when utilizing FP4 quantization, the combined weight of the model parameters, key-value cache, and the Eagle3 draft head cannot fit into 128 gigabytes of memory. To bypass this hardware ceiling, developers can connect two DGX Spark systems using a QSFP cable, doubling the available memory to 256 gigabytes. This setup uses tensor parallelism with a factor of two, splitting model layers across both nodes. The systems communicate intermediate calculation results over a high-bandwidth link using NCCL and OpenMPI, allowing the hardware to function as a unified logical instance to accelerate inference.

Whether these heterogeneous vocabulary algorithms can maintain their lossless efficiency when scaled across multi-node hardware environments like dual-Spark setups remains the next critical test for production-grade AI optimization.