Speculative Decoding (MTP) on an RTX 4090: Measured ~2x
Published 2026-10-14
On an RTX 4090, turning on MTP (multi-token prediction, a form of speculative decoding) took Qwen3.8-27B Q4_K_M from 43.06 to 84.51 tok/s, about 1.96× (llama.cpp, chat-short). Prompt processing got slower: prefill fell from 428.6 to 299.3 tok/s and time to first token rose from 359 to 515 ms. (Measured: MTP off vsMTP on, Oct 14 2026.)
Both runs come from the same rented host (RTX 4090 24GB, AMD EPYC 7642, 252GB RAM), the same llama.cpp build (0.6.0-dev, df18bee) and the same suite (suite-v1). The only difference is the model label: Qwen3.8-27Bvs Qwen3.8-27B-mtp-on. The run pages don't record the MTP settings (draft depth, spec type) or the acceptance rate, so we don't state them.
Measured: MTP off vs on, every workload
| Workload | Decode off, tok/s | Decode on, tok/s | Speedup (calculated from measured decode) | Prefill off → on, tok/s | TTFT off → on | Inter-token p50 off → on | p95 off → on |
|---|---|---|---|---|---|---|---|
| chat-short | 43.06 | 84.51 | 1.96× | 428.6 → 299.3 | 359 → 515 ms | 23.1 → 0.1 ms | 23.2 → 31.4 ms |
| chat-long | 42.63 | 73.10 | 1.71× | 1,915.1 → 1,585.8 | 1,684 → 2,033 ms | 23.4 → 0.2 ms | 23.5 → 31.8 ms |
| concurrent-decode | 43.00 | 87.04 | 2.02× | not measured | not measured | 23.2 → 0.1 ms | 23.3 → 31.5 ms |
| agent-trace | 42.07 | 77.60 | 1.84× | 3,644.0 → 2,912.1 | 629 → 787 ms | 23.3 → 0.1 ms | 23.8 → 32.8 ms |
About the p50/p95 columns: with MTP on, the median gap between tokens drops to about 0.1 ms while the slowest 5% of gaps grow to about 31 ms (from about 23 ms). That pattern is consistent with drafted tokens being accepted in groups and arriving in bursts. It is our interpretation, not a measured acceptance rate.
What MTP and speculative decoding are
A cheap draft guesses several tokens ahead, and the full model checks those guesses in one pass. Accepted guesses come out together. Decode on a GPU is mostly limited by memory bandwidth (every token reads the weights), so checking several tokens per read of the weights is close to free. With MTP, the draft comes from extra prediction heads built into the model rather than from a separate small draft model. More on the two speeds indecode vs prefill tok/s explained.
Where the speedup held
Measured across our four workloads, the gain ranged from 1.71× on chat-long to2.02× on concurrent-decode, with 1.96× on chat-short and 1.84× on agent-trace (calculated). Longer chats gained less in our runs. Our longest workload is chat-long, so don't extrapolate these numbers to 32K-token or longer prompts.
What it costs (measured)
- Slower prefill: about 17–30% slower prompt processing (calculated from the table: 428.6 → 299.3, 1,915.1 → 1,585.8 and 3,644.0 → 2,912.1 tok/s).
- Later first token: about 156–349 ms more time to first token (359 → 515, 1,684 → 2,033 and 629 → 787 ms).
- Burstier output: the p95 gap between tokens rose from about 23 ms to about 31 ms, even though average decode nearly doubled.
- In a chat window this means a slightly later first word, then faster text.
What it means if you're buying a GPU
Reference only, not a controlled test. Our RTX 5090 without MTP measured65.73 tok/s on the same model label, on a different host and llama.cpp build (9725a31). So in our runs, a 4090 with MTP on outran a 5090 with MTP off. We haven't measured MTP on the 5090 yet. These are measured reference points, not a controlled ranking.
See the measured best GPU under $2,000 guide (RTX 3090 vs 4090) and RTX 5090 vs RTX 4090.
How to turn it on
You need a model file that carries MTP heads (a GGUF built with them) and a llama.cpp build that supports MTP drafting. The flags are documented in llama.cpp'sspeculative decoding docs and theserver README. Users report that a draft depth of 3 works well and 4 or more gets slower, with acceptance in the mid-80s percent (user report). That is their setup, not ours; we didn't record these settings.
What we haven't measured yet
- MTP on the RTX 5090 and RTX 3090, on the same build as the 4090 pair. Our MTP-off baselines are 37.65 tok/s on the 3090 and 65.73 on the 5090.
- Longer prompts: 8K, 32K and 128K. Gains already shrank on chat-long.
- Acceptance rate and draft depth. The run pages don't log accepted vs drafted tokens or the draft settings yet.
- Repeats: one run each for off and on. A median of three each would give a range.
- Other approaches: DFlash2, MLX MTP on Apple Silicon, and a separate small draft model.
- Other models, such as the Qwen3.8-35B-A3B MoE.
- An output-equivalence check with the same prompt and seed, on vs off.
Methodology: how the suite measures. All Qwen3.8-27B results are onthe model page; hardware pages: RTX 4090,RTX 3090, RTX 5090. If things still feel slow, readwhy is my local LLM slow?
FAQ
Does MTP make local LLMs faster? In our test, yes, for decode: Qwen3.8-27B Q4_K_M on an RTX 4090 went from 43.06 to 84.51 tok/s (measured, llama.cpp, chat-short).
Is there a downside? Prompt processing slowed: prefill went from 428.6 to 299.3 tok/s and first-token time from 359 to 515 ms in the same comparison (measured).
Does the speedup hold for long chats? It was smaller: 1.71× on our chat-long workload vs 1.96× on chat-short (calculated from measured decode). We haven't tested 32K+ contexts.
Is MTP the same as speculative decoding? MTP is one way to do speculative decoding. The draft comes from extra prediction heads built into the model instead of a separate small draft model.
Will I get 2× on a 3090 or 5090? We haven't measured MTP on those cards yet. Our MTP-off baselines are 37.65 tok/s on the 3090 and 65.73 on the 5090.
Does MTP change output quality? By design, speculative decoding verifies every drafted token against the full model. We haven't run an output-equivalence check.