llama.cpp vs MLX on Qwen3.6-27B: MTP is 1.04x here, not 1.85x
Mihai Perdum
Author
15 min readAugust 10, 2026
Key takeaways
On an M3 Ultra, every out-of-the-box MTP setting in llama.cpp was SLOWER than plain decoding: 0.97x at draft depth 1, down to 0.74x at depth 4.
One config beat baseline: --spec-draft-n-max 3 --spec-draft-p-min 0.8, at 24.02 t/s vs 23.08. That is 1.04x, against the 1.85x the merge PR reports on CUDA.
Each drafted token costs a flat ~30 ms on this hardware, regardless of depth. That is ~70% of a full 27B decode step to run a single extra layer.
mlx-lm does not ignore the MTP head, it deletes it: sanitize() drops every mtp.* key, so all three MLX conversions on my disk have 0 MTP tensors while their configs still advertise mtp_num_hidden_layers: 1.
MLX with no MTP at all ran 27.06 t/s, 13% faster than the best-tuned llama.cpp MTP config, while holding about 14% more language-model bytes.
On a contended machine the same comparison inverted and MTP looked like a 1.20x win. Check your bandwidth before you trust a speedup.
I went looking for a speedup I had been told was free.
Multi-token prediction is the 2026 version of speculative decoding that does not need a second model. The lab trains the model with an extra prediction head, the head drafts the next few tokens, the main model verifies them in one batch, and you keep whatever it got right. No draft model to pick, no vocabulary mismatch to reconcile, nothing extra to download. The head ships inside the same file.
It landed in llama.cpp in May. The merge PR reports about 1.85x on Qwen3.6 27B. The model card for the exact GGUF I had on disk claims 1.66x. So the question I actually cared about was narrow and easy to state: on Apple Silicon, with the same model, does the runtime that can use the MTP head beat the runtime that cannot?
The answer turned out to be no, in both directions at once. llama.cpp can use the head and mostly loses time doing it. MLX cannot use the head, throws the weights away at load, and is still the fastest way to run that model on this machine.
Here is everything I measured and the arithmetic that explains it.
The two runtimes, and where MTP actually stands
Two separate pull requests, two different fates. I checked both against the source rather than from memory, because this area moves faster than anyone's recollection of it.
llama.cpp — merged.PR #22673, "llama + spec: MTP Support", merged 2026-05-16. It adds --spec-type draft-mtp and reuses the existing --spec-draft-n-max knob. The MTP head lives in the same GGUF as its own context and KV-cache. The PR description reports roughly 1.85x on Qwen3.6 27B with 3 draft tokens at about 72% acceptance.
mlx-lm — still open.PR #990, "Native MTP speculative decoding (Qwen3.5/3.6 reference implementation)", last updated 2026-07-16 and still open when I checked the API on 2026-08-10. It reports ~1.5x on dense models at 80.6% acceptance. A maintainer's position in the thread is that a feature of that size takes real review time, which is fair, and it is why Apple Silicon users do not have this yet.
So on paper: llama.cpp has the feature, MLX does not. That framing turns out to understate what MLX does.
MLX does not skip the MTP head, it deletes it
This is the part I did not expect, and it is worth being precise about because "MLX does not support MTP" and "MLX destroys the MTP weights" are different claims and only one of them is true.
Start upstream. Qwen/Qwen3.6-27B ships its MTP head in the open, 15 tensors out of 1199:
bash
1curl-sL https://huggingface.co/Qwen/Qwen3.6-27B/raw/main/model.safetensors.index.json \2| python3 -c'import json,sys; wm=json.load(sys.stdin)["weight_map"]; \
3 mtp=[k for k in wm if "mtp" in k]; print(len(wm), "tensors,", len(mtp), "mtp"); \
4 print("\n".join(sorted(mtp)))'
Note the shape of that list. Attention projections with their norms, a full MLP, two layernorms. The MTP head is not a little linear probe bolted onto the logits. It is a complete extra transformer layer plus a fusion projection. Hold that thought, it is the whole explanation for the timings later.
Now the MLX side. In mlx_lm/models/qwen3_5.py, the model's sanitize() method contains this:
python
1defsanitize(self, weights):2 has_mtp_weights =any("mtp."in k for k in weights)3 has_unsanitized_conv1d =any(4"conv1d.weight"in k and v.shape[-1]!=1for k, v in weights.items()5)6 should_shift_norm_weights = has_mtp_weights or has_unsanitized_conv1d
7 weights ={k: v for k, v in weights.items()if"mtp."notin k}
Read those five lines in order, because the sequence is almost funny. It detects that MTP weights are present. It uses their presence as a format flag, since a checkpoint carrying an MTP head also uses a different layernorm convention that needs shifting. Then, having used them as a signal, it filters every one of them out.
sanitize() is not some optional path. mlx_lm/utils.py calls it on every load:
and mlx_lm/convert.py builds its output by calling that same load(). So the strip happens both when you load a model and when you convert one.
I did not want to take the code reading on trust, so I ran the real upstream manifest through the real function. No weights needed, the filter works on keys:
python
1import json, mlx.core as mx
2from mlx_lm.models import qwen3_5
34cfg = json.load(open("q36_config.json"))# Qwen/Qwen3.6-27B config.json5args = qwen3_5.ModelArgs.from_dict(cfg["text_config"])6model = qwen3_5.Model(args)78keys =list(json.load(open("idx_q36.json"))["weight_map"].keys())# the real 1199 keys9out = model.sanitize({k: mx.zeros((1,))for k in keys})1011print("in :",len(keys),"keys,",len([k for k in keys if"mtp"in k.lower()]),"mtp")12print("out:",len(out),"keys,",len([k for k in out if"mtp"in k.lower()]),"mtp")
text
1in : 1199 keys, 15 mtp
2out: 851 keys, 0 mtp
Fifteen in, zero out.
Note
Do not read 1199 → 851 as "348 tensors deleted". sanitize() also renames keys (model.language_model.layers.0… becomes language_model.model.layers.0…), so the two key sets barely overlap by construction. The only claim that survives inspection is the one I actually tested: of the 15 keys containing mtp, none survive.
And this is not theoretical for anyone using MLX today. Three Qwen3.6-class MLX conversions were already sitting on my disk, from three unrelated publishers. All three tell the same story:
Every one of them advertises an MTP head in config.json that is not present in the weights. If PR #990 merges tomorrow, none of these files can use it. The whole MLX ecosystem for this model family will need reconverting from source.
One correction to a claim you will see made loosely: mlx-lm does have speculative decoding. mlx_lm.generate takes --draft-model and --num-draft-tokens, and it works. What it lacks is the native MTP path, the one where the draft head rides along inside the same file. If you want speculative decoding in MLX today you must find and load a second, smaller model.
The rig
Everything below ran on one machine, and I am going to be tedious about it because the last section of this article is about how easily these numbers lie.
Mac Studio, Apple M3 Ultra, 28 CPU cores (20 performance, 8 efficiency), 60 GPU cores, 96 GB unified memory, macOS 26.6 (build 25G72).
If you use LM Studio, note that it bundles its own builds and they are not these: llama.cpp-mac-arm64-apple-metal-advsimd@2.27.1 and mlx-llm-mac-arm64-apple-metal-advsimd@1.11.0. Different versions, potentially different behaviour. Say which one you measured.
One line from llama.cpp's Metal init that Apple Silicon owners should know exists:
text
1ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
The M3 Ultra does not get that path. That is a statement about this box, not a prediction about yours.
For the model I wanted the same weights in both runtimes. Qwopus3.6-27B-Coder, a 27.32 B coding fine-tune of Qwen3.6-27B, exists as both a GGUF with the MTP head intact and an MLX conversion:
llama.cpp
MLX
artifact
Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF
mlx-community/Qwopus3.6-27B-Coder-6bit
quantisation
Q5_K_M
6-bit affine, group 64
language-model bytes
19.22 GB
21.86 GB
vision tower
separate mmproj, not loaded
0.92 GB, included in the files
resident
18329.72 MiB GPU + 833.59 MiB CPU
22.15 GB peak
MTP head
present, 15 tensors, 304,601,088 B
absent
7 rows × 3 columnsHeader row enabled
Both are multimodal checkpoints, so I split the tensors by role before comparing anything. The text GGUF carries no vision weights at all (they live in a separate mmproj-F32.gguf that I never loaded), while the MLX conversion ships a 0.92 GB vision tower inside its safetensors. The row above is language-model weights only, which is the part that gets read on every decode step.
The quantisations are not identical and I am not going to pretend they are. Q5_K_M works out to about 5.6 bits per weight here, the MLX 6-bit conversion to about 6.4. That asymmetry runs against MLX: it is holding about 14% more language-model bytes than llama.cpp is. Keep that in mind when MLX wins anyway.
Benchmark protocol, identical for every llama.cpp row: llama-server with -ngl 99 -c 8192, three fixed prompts, n_predict: 256, temperature: 0, seed: 42, cache_prompt: false. Aggregate throughput is total predicted tokens over total predicted milliseconds, taken from the server's own timings object rather than a stopwatch. MLX ran the same three prompts at 256 tokens with a temperature-0 sampler.
Temperature 0 with a fixed seed matters more than it looks. It makes the token stream deterministic, which means the draft-acceptance counts are reproducible to the token, and I can compare acceptance across runs that differ wildly in wall-clock.
llama.cpp: every default MTP setting was slower
First the thing you get for free without asking, because it is a trap. Load an MTP GGUF normally and llama.cpp tells you exactly what it thinks of the head:
Fifteen tensors, 304,601,088 bytes, 290.5 MiB, enumerated and then dropped on the floor. MTP is off by default. If you downloaded a file with MTP in the name and changed nothing else, you are carrying the head and not using it. You have to ask:
1common_speculative_init_result: creating MTP draft context against the target model
So I asked, at every draft depth from 1 to 4:
config
tokens/sec
vs baseline
draft acceptance
no MTP
23.08
1.00x
—
--spec-draft-n-max 1
22.34
0.97x
79.3% (338/426)
--spec-draft-n-max 2
21.45
0.93x
69.0% (442/641)
--spec-draft-n-max 3
19.60
0.85x
59.2% (488/824)
--spec-draft-n-max 4
17.15
0.74x
49.2% (505/1027)
6 rows × 4 columnsHeader row enabled
Every single one is a regression, and it gets monotonically worse the more you draft. At the depth the merge PR recommends, I lost 15% of my throughput.
The acceptance column is not the problem. 59.2% at depth 3 is in the same neighbourhood as the 72% the PR reports, and the shape is exactly right: the deeper you draft off a single-layer head running autoregressively, the more it drifts, so acceptance falls from 79.3% to 49.2%. The head works. The head is doing its job. The economics around it are what fail.
Where the time goes: ~30 ms per drafted token
Speculative decoding is a bet with a fixed structure. Each step you pay for n draft generations plus one verification pass, and in exchange you get somewhere between 1 and n+1 tokens. It wins when drafting is cheap relative to a decode step. So I worked out what a step actually cost.
With temperature 0 the arithmetic is clean. Every step yields one guaranteed token plus however many drafts were accepted, so the number of steps is just tokens - accepted:
config
tokens
accepted
steps
ms/step
tokens/step
no MTP
768
—
768
43.3
1.00
n-max 1
768
338
430
80.0
1.79
n-max 2
768
442
326
109.8
2.36
n-max 3
768
488
280
140.0
2.74
n-max 4
768
505
263
170.2
2.92
6 rows × 6 columnsHeader row enabled
Now take the differences down that ms/step column:
text
1depth 1 -> 2 : +29.9 ms
2depth 2 -> 3 : +30.1 ms
3depth 3 -> 4 : +30.3 ms
Every additional drafted token costs a flat 30 ms. Three measurements, spread of 0.4 ms. That is about as clean a linear cost as you will ever measure on a real machine, and it is the entire article in one number.
Compare it to the thing it is supposed to be cheaper than. A full decode step through all 64 layers of a 27B model takes 43.3 ms. Running one extra layer to draft one token costs 30 ms, which is 69% of a full forward pass.
That sets the bar. Every draft costs 30 ms whether it survives or not, and each one that survives saves a 43.3 ms decode step, so drafting only pays when acceptance clears 30 / 43.3, which is 69%. Look back at the acceptance column with that number in hand. Depth 3 runs at 59.2% and depth 4 at 49.2%, both well under the bar, which is exactly why they lose 15% and 26%. Depth 1 runs at 79.3%, comfortably over the bar, and still loses 3%. The reason is in the step times: a depth-1 step should cost 43.3 + 30 = 73.3 ms and it measures 80.0. There is about 7 ms per step that is neither the draft nor a plain decode, and it is enough to eat a margin that was only ever 10 points wide.
Why is one layer so expensive? Not bandwidth, as far as I can account for it. A draft step needs the 65th layer (290.5 MiB) and the output projection over the 248,320-token vocabulary (output.weight, 1.04 GB, which the head shares rather than duplicating). Call it 1.35 GB. The main model moves about 18.3 GB in 43.3 ms, so it is running at roughly 420 GB/s. At that rate 1.35 GB should take 3.2 ms. It takes 30.
Warning
That last paragraph is arithmetic, not instrumentation. I did not profile inside llama.cpp to see where the 27 ms difference goes, and I am not going to assert a cause I did not measure. What I can say is that the draft step is nine times less efficient per byte than the main model's own forward pass, and that its cost does not scale with anything I varied. Fixed per-call overhead fits that shape. So would a draft path that quietly does more work than the head alone. I do not know which, and the log line "creating MTP draft context against the target model" is suggestive without being conclusive.
The one setting that beats baseline
If the problem is paying 30 ms for drafts that get rejected, the fix is to stop drafting when the head is not confident. llama.cpp has exactly that knob and it defaults to off: --spec-draft-p-min, which stops the draft loop once token probability drops below a threshold.
config
tokens/sec
vs baseline
acceptance
no MTP
23.08
1.00x
—
-n-max 3 --p-min 0.8
24.02
1.04x
94.9% (353/372)
-n-max 2 --p-min 0.9
23.74
1.03x
97.5% (275/282)
-n-max 3 --p-min 0.9
23.65
1.02x
96.9% (286/295)
-n-max 6 --p-min 0.9
23.10
1.00x
95.8% (298/311)
-n-max 4 --p-min 0.95
21.92
0.95x
95.8% (251/262)
7 rows × 4 columnsHeader row enabled
So it can be rescued. Gate the drafting hard and acceptance jumps into the mid-90s, wasted draft calls mostly disappear, and the best configuration I found finally clears baseline:
24.02 t/s against 23.08. A 4% gain, out of a feature the merge PR measures at 85% on CUDA.
And it is genuinely delicate. Push to --p-min 0.95 at depth 4 and you are back to a 5% loss despite 95.8% acceptance: the gate fires so aggressively that only 262 drafts survive across 517 steps, and the fixed cost of running the machinery outweighs the handful of tokens it buys. There is no broad plateau of good settings here. There is a narrow ridge, and everything either side of it is worse than not turning the feature on.
If you drive this through LM Studio rather than llama.cpp directly, the same controls are there under different names:
--speculative-draft-min-continue-probability is the one that matters. Without it you are on the regression table, not the rescue table.
And then MLX, which cannot do any of this, wins
Same model, same three prompts, same 256 tokens, no MTP head in the file at all:
text
1tg 256 tok in 9.56s = 26.79 t/s
2tg 256 tok in 9.40s = 27.22 t/s
3tg 256 tok in 9.43s = 27.16 t/s
4AGG mlx-lm: 768 tok / 28.4s = 27.06 t/s
5peak memory: 22.15 GB
Put the whole board together:
runtime
configuration
tokens/sec
mlx-lm 0.31.3
6-bit, no MTP available
27.06
llama.cpp b10330
Q5_K_M, MTP tuned (n 3, p-min 0.8)
24.02
llama.cpp b10330
Q5_K_M, no MTP
23.08
llama.cpp b10330
Q5_K_M, MTP at recommended depth 3
19.60
5 rows × 3 columnsHeader row enabled
MLX is 13% faster than the best MTP configuration I could find, 17% faster than llama.cpp's plain baseline, and 38% faster than llama.cpp running MTP the way the PR suggests. It does this while holding about 14% more language-model bytes, because its quantisation is coarser. Correct for that and the gap widens.
That is the finding I did not go looking for. The interesting difference between these two runtimes on Apple Silicon right now is not the feature one of them has. It is the per-token efficiency of the plain decode loop, and MLX is simply better at it on this hardware. A 4% speculative-decoding win is noise next to a 17% baseline gap.
One incidental thing worth knowing if you are sizing context for this model family: only 16 of the 64 layers use ordinary KV attention. The rest are linear-attention layers carrying a recurrent state.
256 MiB of KV for 4k context on a 27B model. Long context is cheap here in a way it is not on a conventional transformer.
The run where MTP looked like a 1.20x win
I nearly published the opposite of this article, so this section is the important one.
My first pass ran with LM Studio in the background holding a 27B model resident, 38 to 44 GB of it. Its own status field claimed the model was generating, but the process sat at ~5% CPU with no server log written in hours, so I read it as a session someone had walked away from and left the weights loaded. It looked harmless either way. Here is what I measured:
config
contended
quiet
no MTP
11.68 t/s
23.08 t/s
--spec-draft-n-max 3
14.06 t/s
19.60 t/s
verdict
MTP wins, 1.20x
MTP loses, 0.85x
4 rows × 3 columnsHeader row enabled
Same binary, same model, same prompts, same seed. The contention did not just add noise, it inverted the sign of the result. The baseline degraded further than the MTP path did, so MTP came out ahead, and I would have written up a 1.20x speedup that does not exist.
The draft acceptance rate was 59.2% in both columns. Identical, 488 accepted of 824 drafted, to the token. Ratios of a deterministic token stream survive contention perfectly. Timings do not. If I had only reported acceptance I would have looked rigorous and been wrong.
What caught it was a bandwidth check rather than suspicion. An M=1 bf16 GEMV on this machine should come close to its ~819 GB/s spec:
python
1import mlx.core as mx, time
23N =81924W = mx.random.normal((N, N)).astype(mx.bfloat16); mx.eval(W)5x = mx.random.normal((1, N)).astype(mx.bfloat16); mx.eval(x)67defgemv(reps):8 ys =[x @ W for _ inrange(reps)]# independent, so they pipeline9 mx.eval(ys); mx.synchronize()1011gemv(5)# warm up12best =float("inf")13for _ inrange(4):14 t0 = time.perf_counter(); gemv(40); best =min(best,(time.perf_counter()- t0)/40)1516print(f"{N * N *2/ best /1e9:.0f} GB/s")
Contended it read 540 GB/s. Quiet it read 667 GB/s. That gap was the tell.
Two smaller traps from the same afternoon, both of which cost me a run:
Chaining the matmuls as y = y @ W instead of collecting independent results measured 133 GB/s, against 540 GB/s for the independent form on the same machine in the same state. Four times too low, and nothing to do with contention: dependent operations serialise and you end up timing kernel launch latency rather than memory. Separately, running llama-cli with -v to capture metadata dropped generation from 13.1 to 11.0 t/s, which is why the verbose logs quoted in this article come from different runs than the timings do.
The discipline that survives all this is dull and worth stating plainly. Measure the machine before you measure the software, run the baseline again at the end of the session (mine closed at 23.06 t/s against an opening 23.08, so nothing drifted), and treat any result that agrees with the marketing as the one most in need of a second look.
What to actually do
1
Check whether your GGUF even has the head
run llama-server once and grep the log for nextn. If you see model has unused tensor blk.NN.nextn.* -- ignoring, the head is in your file and switched off.
2
Measure your baseline first, on a quiet machine
close other model runners, confirm nothing is resident, and record plain decode throughput with a fixed seed at temperature 0. Without this number nothing else you measure means anything.
3
Do not enable MTP with defaults
--spec-type draft-mtp --spec-draft-n-max 3 on its own cost me 15% on Apple Silicon. If you turn it on, turn on the gate with it.
4
Start at --spec-draft-n-max 3 --spec-draft-p-min 0.8
the only region I found that beats baseline, and it beats it by 4%. In LM Studio the equivalent is --speculative-draft-mtp plus --speculative-draft-min-continue-probability 0.8.
5
Re-run the baseline afterwards
if it no longer matches the opening number, throw the session away rather than the hypothesis.
6
On Apple Silicon, try MLX before you tune any of this
for this model family it was faster with no speculative decoding at all than llama.cpp was with the feature tuned.
What I did not test
The honest boundaries of the above, because the comparison has real limits.
This is one model family on one machine. Qwen3.6 is a hybrid linear-attention architecture with only 16 conventional KV layers out of 64, and I would not assume any of these ratios transfer to a dense transformer or to an MoE. The merge PR's 1.85x was measured on CUDA, where the balance between a draft call and a decode step is completely different, and nothing here contradicts it on that hardware.
The two quantisations are not matched. Q5_K_M against MLX 6-bit affine is close enough to compare directions but not close enough to quote a precise ratio, and I did not build a matched pair because doing it properly means converting both from bf16 source.
Three prompts at 256 tokens is a short-generation, short-context workload. Acceptance rates tend to be higher on repetitive or structured output, so a long code-generation run may treat MTP more kindly than my mix did. I also did not test batched or concurrent serving, where the verification pass amortises across requests and the arithmetic changes completely.
And I did not test PR #990. If it merges, MLX gets native MTP, and the first thing that will need to happen is a wave of reconversions, because every MLX build of this model family currently in circulation has already had the head stripped out.
The thing I would most like to be wrong about is the 30 ms. If someone profiles that draft path on Metal and finds it is fixed overhead rather than real work, this feature goes from a 4% curiosity to the speedup it was supposed to be, on every Mac running llama.cpp. Has anyone put an instrument on it?
If it is memory rather than throughput you are sizing for, I have since measured that on this same machine — Metal hands the GPU 77.76 GiB of the 96 GB installed, not the 75% everyone repeats, and the cache arithmetic has changed completely. That is in 96 GB is 77.76 GiB: the real memory ceiling on an M3 Ultra.
Metal will not give you the RAM on the box, and the number it does give is not the 75% everyone repeats. I measured the three ceilings on a 96 GB Mac Studio, then measured what modern hybrid-attention models actually spend against them — including a Gemma 4 cache that quietly holds three times its own sliding window.
