MTP for Qwen3.8-Flash-Next Landed in llama.cpp, and It Nearly Doubles Decode Speed on the GB10
This feature merged, so I tested it on the GB10. Multi-token prediction (MTP) for the Qwen3.8-Flash-Next architecture is now in llama.cpp master. I rebuilt on the current commit (bed0a8566, build 11370) and ran it against the same 125B mixture-of-experts model I already serve every day. Short version: decode went from 18.9 to about 32 tokens per second, and getting there cost me half my context window and one very bad morning.
The numbers
The model is Qwen3.8-Flash-Next at 4.27 bits per weight, 89GB on disk, 6B active parameters, running entirely on the GPU of a DGX Spark with 121GB of unified memory. Every number below is the median of five runs at temperature 0, 256 output tokens, through the normal chat endpoint.
Draft length barely matters once you are past 2. The 2 to 5 spread is about 6 percent, so I picked 4 and moved on.
That split is the whole story of speculative decoding. On code and JSON, the MTP head guesses the next tokens correctly most of the time, so you get a big win. On open prose there is more entropy, the head is right about half the time, and you still win, just by less. Nothing got slower.
The catch: memory
The model alone is 89GB and the MTP head adds 2.6GB. On a unified memory box, that leaves very little for the KV cache, the vision projector, and the rest of the operating system. My first attempt used my production settings (262k context, vision on) and hung the whole machine for about four hours until I power cycled it. The kernel never OOM-killed anything. Memory just sat at 6 percent free while the GPU driver threw allocation errors and every watchdog on the box timed out.
| Config | Min free | Result |
|---|---|---|
| 32k ctx, no vision | 17 to 18GB | fits |
| 131k ctx, vision, 1 slot, 1GB prompt cache | 15.1GB | fits |
| 131k ctx, vision, 4 slots, default cache | 12.0GB | tripped the kill switch |
| 262k ctx, vision, MTP | 11.1 to 11.8GB | does not fit |
Two defaults were quietly eating the margin. llama-server starts with four slots, and each carries its own recurrent state. It also keeps up to 8GB of prompt cache in host memory. Dropping to one slot and a 1GB cache is what made 131k context fit with room to spare.
The settings I am running
llama-server \
--model Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf \
--mmproj mmproj-Qwen3.8-Flash-Next-F16.gguf \
--spec-draft-model mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf \
--spec-type draft-mtp --spec-draft-n-max 4 \
-ngl 999 -c 131072 -np 1 --cache-ram 1024 --jinja
Live on the box with these flags it holds 32.1 tok/s with 16GB still free. I also stopped Ollama entirely so nothing else competes for the pool.
What I would tell you before you try this
- Free the memory first. After a crash or a long session the GB10 can hold tens of GB of stuck memory that no process owns.
sync; echo 3 | sudo tee /proc/sys/vm/drop_cachesgot me back from 57GB used to 6GB. - Make the OOM killer act on RAM alone. The default earlyoom setup only fires when memory and swap are both low, and swap was 80 percent empty while the box was dying. I changed it to trigger on memory only.
- Test under a leash. I ran every config with a watchdog that stopped the server below 12GB free. It tripped three times. Each of those would have been another reboot.
- Not every MTP head loads. I have two on disk, and only one of them works with this build.
I ran this with Claude doing the legwork, and I checked the numbers myself. The 262k context is gone for now. I would trade that for a 1.7x speedup every time, and when the next build trims the memory footprint I will test again.