vLLM Enables Speculative Decoding on AMD Instinct GPUs

Serving large language models efficiently remains a core engineering challenge as request volumes grow.

Standard autoregressive decoding generates one token per model pass, forcing a strict left‑to‑right progression.

This token‑by‑token loop can dominate latency during long generations, limiting overall throughput.

Baseline Autoregressive Decoding

In the baseline workflow, each decode step consumes the current context, produces a single token, and appends that token to the input for the next step.

Four output tokens therefore require four separate model invocations, as illustrated in the source description.

The simplicity of this loop makes it reliable, but the one‑token‑per‑step cadence creates a performance bottleneck.

Speculative Decoding Mechanism

Speculative decoding introduces a draft‑and‑verify stage that separates proposal from final verification.

A lightweight draft component first proposes multiple candidate tokens for future positions.

The original model, retained as the target model, then evaluates the entire candidate sequence in a single verification pass.

Accepted draft tokens are committed, allowing several output tokens to be produced from one target‑model evaluation.

The approach preserves the target model’s output behavior while reducing the number of expensive passes.

“Speculative decoding does not replace the original model.”

“Instead, it keeps the original model as the target model, which remains responsible for the final output, and adds a faster proposal stage in front of it.”

Five draft strategies were examined: native MTP, Gemma 4 MTP, EAGLE‑3, DFlash, and DSpark.

These methods vary in how the draft component receives information from the target model and whether tokens are generated sequentially, autoregressively, in parallel, or via hybrid schemes.

Evaluation on AMD Instinct GPUs

Experiments were conducted on AMD Instinct MI300X and MI355X GPUs using the ROCm™ open software platform.

The test environment measured output‑token throughput across different drafting methods, proposal lengths, model families, draft checkpoints, workloads, and acceptance behaviors.

Results showed that speculative decoding’s impact on throughput was not uniform; performance gains depended on the combination of draft method and hardware.

Longer proposal lengths sometimes improved throughput but also increased verification cost, creating a trade‑off.

Model families with higher acceptance rates benefited more, as more draft tokens could be committed per verification pass.

Workload characteristics, such as request batch size and sequence length, further influenced the observed speedups.

Practical tuning recommendations included selecting a draft method aligned with the target model’s architecture and adjusting proposal length to match GPU memory constraints.

Observability considerations highlighted the need to monitor acceptance ratios and verification latency to maintain stable service levels.

Overall, the study demonstrates that speculative decoding can be integrated into vLLM deployments on AMD hardware, offering a pathway to higher serving efficiency when configured appropriately.

Why This Matters