llama.cpp vs MLX on Qwen3.6-27B: MTP is 1.04x here, not 1.85x

llama.cpp vs MLX on Qwen3.6-27B: MTP is 1.04x here, not 1.85x

Mihai Perdum
Mihai Perdum
Author

15 min readAugust 10, 2026

Key takeaways

  • On an M3 Ultra, every out-of-the-box MTP setting in llama.cpp was SLOWER than plain decoding: 0.97x at draft depth 1, down to 0.74x at depth 4.
  • One config beat baseline: --spec-draft-n-max 3 --spec-draft-p-min 0.8, at 24.02 t/s vs 23.08. That is 1.04x, against the 1.85x the merge PR reports on CUDA.
  • Each drafted token costs a flat ~30 ms on this hardware, regardless of depth. That is ~70% of a full 27B decode step to run a single extra layer.
  • mlx-lm does not ignore the MTP head, it deletes it: sanitize() drops every mtp.* key, so all three MLX conversions on my disk have 0 MTP tensors while their configs still advertise mtp_num_hidden_layers: 1.
  • MLX with no MTP at all ran 27.06 t/s, 13% faster than the best-tuned llama.cpp MTP config, while holding about 14% more language-model bytes.
  • On a contended machine the same comparison inverted and MTP looked like a 1.20x win. Check your bandwidth before you trust a speedup.

I went looking for a speedup I had been told was free.

Multi-token prediction is the 2026 version of speculative decoding that does not need a second model. The lab trains the model with an extra prediction head, the head drafts the next few tokens, the main model verifies them in one batch, and you keep whatever it got right. No draft model to pick, no vocabulary mismatch to reconcile, nothing extra to download. The head ships inside the same file.

It landed in llama.cpp in May. The merge PR reports about 1.85x on Qwen3.6 27B. The model card for the exact GGUF I had on disk claims 1.66x. So the question I actually cared about was narrow and easy to state: on Apple Silicon, with the same model, does the runtime that can use the MTP head beat the runtime that cannot?

The answer turned out to be no, in both directions at once. llama.cpp can use the head and mostly loses time doing it. mlx-lm cannot use the head, throws the weights away at load, and was still the fastest thing I ran on this machine.

Here is everything I measured and the arithmetic that explains it.

Correction, 24 October 2026: I wrote that Apple Silicon users don't have MTP on MLX yet, and that every MLX build of this model family has the head stripped out. Both were wrong, and both were already wrong when I published. Upstream mlx-lm still deletes the head, and that part of my measurement stands. But oMLX shipped native MTP for Qwen3.5 and 3.6 in May, Ollama's MLX engine started using the head for Qwen3.5 in August, and builds that keep the head in a separate file exist. The measurements below haven't changed. They're from August 2026, on llama.cpp b10330 and mlx-lm 0.31.3.

The two runtimes, and where MTP actually stands

Two separate pull requests, two different fates. I checked both against the source rather than from memory, because this area moves faster than anyone's recollection of it.

llama.cpp — merged. PR #22673, "llama + spec: MTP Support", merged 2026-05-16. It adds --spec-type draft-mtp and reuses the existing --spec-draft-n-max knob. The MTP head lives in the same GGUF as its own context and KV-cache. The PR description reports roughly 1.85x on Qwen3.6 27B with 3 draft tokens at about 72% acceptance.

mlx-lm — still open. PR #990, "Native MTP speculative decoding (Qwen3.5/3.6 reference implementation)", was still open when I checked again on 24 October 2026. Its description now reports 1.50x to 1.57x on Qwen3.6-27B 4-bit on an M4 Pro, at 80.5% to 88.3% acceptance. On 21 August the maintainer asked for changes: he wants the MTP head exposed as a draft model for the existing speculative decoding path, and the blocker he named, a trim for ArraysCache, is what his own open PR #1821 is working on. So upstream mlx-lm still has no MTP. Other MLX runtimes do.

So on paper: llama.cpp has the feature, upstream mlx-lm does not. That framing turns out to understate what mlx-lm does.

mlx-lm does not skip the MTP head, it deletes it

I did not expect this, and it is worth being exact. "mlx-lm ignores the MTP head" and "mlx-lm destroys the MTP weights" are different claims, and only the second one is true.

Start upstream. Qwen/Qwen3.6-27B ships its MTP head in the open, 15 tensors out of 1199:

bash
1curl -sL https://huggingface.co/Qwen/Qwen3.6-27B/raw/main/model.safetensors.index.json \
2  | python3 -c 'import json,sys; wm=json.load(sys.stdin)["weight_map"]; \
3      mtp=[k for k in wm if "mtp" in k]; print(len(wm), "tensors,", len(mtp), "mtp"); \
4      print("\n".join(sorted(mtp)))'
text
11199 tensors, 15 mtp
2mtp.fc.weight
3mtp.layers.0.input_layernorm.weight
4mtp.layers.0.mlp.down_proj.weight
5mtp.layers.0.mlp.gate_proj.weight
6mtp.layers.0.mlp.up_proj.weight
7mtp.layers.0.post_attention_layernorm.weight
8mtp.layers.0.self_attn.k_norm.weight
9mtp.layers.0.self_attn.k_proj.weight
10mtp.layers.0.self_attn.o_proj.weight
11mtp.layers.0.self_attn.q_norm.weight
12mtp.layers.0.self_attn.q_proj.weight
13mtp.layers.0.self_attn.v_proj.weight
14mtp.norm.weight
15mtp.pre_fc_norm_embedding.weight
16mtp.pre_fc_norm_hidden.weight

Note the shape of that list. Attention projections with their norms, a full MLP, two layernorms. The MTP head is not a little linear probe bolted onto the logits. It is a complete extra transformer layer plus a fusion projection. Hold that thought, it is the whole explanation for the timings later.

Now the MLX side. In mlx_lm/models/qwen3_5.py, the model's sanitize() method contains this:

python
1def sanitize(self, weights):
2    has_mtp_weights = any("mtp." in k for k in weights)
3    has_unsanitized_conv1d = any(
4        "conv1d.weight" in k and v.shape[-1] != 1 for k, v in weights.items()
5    )
6    should_shift_norm_weights = has_mtp_weights or has_unsanitized_conv1d
7    weights = {k: v for k, v in weights.items() if "mtp." not in k}

Read those five lines in order, because the sequence is almost funny. It detects that MTP weights are present. It uses their presence as a format flag, since a checkpoint carrying an MTP head also uses a different layernorm convention that needs shifting. Then, having used them as a signal, it filters every one of them out. That was mlx-lm 0.31.x, the version I ran. PR #1623, merged on 18 August and released in v0.32.0 on 15 October, stopped using the MTP keys as the norm signal. The line that drops every mtp.* key is still on main today.

sanitize() is not some optional path. mlx_lm/utils.py calls it on every load:

python
1if hasattr(model, "sanitize"):
2    weights = model.sanitize(weights)

and mlx_lm/convert.py builds its output by calling that same load(). So the strip happens both when you load a model and when you convert one.

I did not want to take the code reading on trust, so I ran the real upstream manifest through the real function. No weights needed, the filter works on keys:

python
1import json, mlx.core as mx
2from mlx_lm.models import qwen3_5
3
4cfg  = json.load(open("q36_config.json"))          # Qwen/Qwen3.6-27B config.json
5args = qwen3_5.ModelArgs.from_dict(cfg["text_config"])
6model = qwen3_5.Model(args)
7
8keys = list(json.load(open("idx_q36.json"))["weight_map"].keys())   # the real 1199 keys
9out  = model.sanitize({k: mx.zeros((1,)) for k in keys})
10
11print("in :", len(keys), "keys,", len([k for k in keys if "mtp" in k.lower()]), "mtp")
12print("out:", len(out),  "keys,", len([k for k in out  if "mtp" in k.lower()]), "mtp")
text
1in : 1199 keys, 15 mtp
2out: 851 keys, 0 mtp

Fifteen in, zero out.

Note

Do not read 1199 → 851 as "348 tensors deleted". sanitize() also renames keys (model.language_model.layers.0… becomes language_model.model.layers.0…), so the two key sets barely overlap by construction. The only claim that survives inspection is the one I actually tested: of the 15 keys containing mtp, none survive.

And this is not theoretical for anyone using MLX today. Three Qwen3.6-class MLX conversions were already sitting on my disk, from three unrelated publishers. All three tell the same story:

MLX conversion

tensors

MTP tensors

config says

lmstudio-community/Qwen3.6-27B-MLX-8bit

2180

0

mtp_num_hidden_layers: 1

nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-qx86-hi-mlx

2180

0

mtp_num_hidden_layers: 1

mlx-community/Qwopus3.6-27B-Coder-6bit

2180

0

mtp_num_hidden_layers: 1

All three advertise an MTP head in config.json that is not present in the weights. If PR #990 merges, none of these three files can use it, and they'd need reconverting from source with the head kept. Not every MLX build is like this: some publishers ship the head as a separate file next to the weights, which I didn't know when I first wrote this.

One correction to a claim you will see made loosely: mlx-lm does have speculative decoding. mlx_lm.generate takes --draft-model and --num-draft-tokens, and it works. It has no native MTP path, where the draft head sits inside the same file. If you want speculative decoding in mlx-lm today you must find and load a second, smaller model.

The rig

Everything below ran on one machine, and I am going to be tedious about it because the last section of this article is about how easily these numbers lie.

Mac Studio, Apple M3 Ultra, 28 CPU cores (20 performance, 8 efficiency), 60 GPU cores, 96 GB unified memory, macOS 26.6 (build 25G72).

text
1llama.cpp   b10330 (687e77892), Homebrew bottle, Metal backend
2mlx         0.32.0
3mlx-lm      0.31.3          (Python 3.13 venv)

If you use LM Studio, note that it bundles its own builds and they are not these: llama.cpp-mac-arm64-apple-metal-advsimd@2.27.1 and mlx-llm-mac-arm64-apple-metal-advsimd@1.11.0. Different versions, potentially different behaviour. Say which one you measured.

One line from llama.cpp's Metal init that Apple Silicon owners should know exists:

text
1ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices

The M3 Ultra does not get that path. That is a statement about this box, not a prediction about yours.

For the model I wanted the same weights in both runtimes. Qwopus3.6-27B-Coder, a 27.32 B coding fine-tune of Qwen3.6-27B, exists as both a GGUF with the MTP head intact and an MLX conversion:

llama.cpp

MLX

artifact

Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF

mlx-community/Qwopus3.6-27B-Coder-6bit

quantisation

Q5_K_M

6-bit affine, group 64

language-model bytes

19.22 GB

21.86 GB

vision tower

separate mmproj, not loaded

0.92 GB, included in the files

resident

18329.72 MiB GPU + 833.59 MiB CPU

22.15 GB peak

MTP head

present, 15 tensors, 304,601,088 B

absent

Both are multimodal checkpoints, so I split the tensors by role before comparing anything. The text GGUF carries no vision weights at all (they live in a separate mmproj-F32.gguf that I never loaded), while the MLX conversion ships a 0.92 GB vision tower inside its safetensors. The row above is language-model weights only, which is the part that gets read on every decode step.

The quantisations are not identical and I am not going to pretend they are. Q5_K_M works out to about 5.6 bits per weight here, the MLX 6-bit conversion to about 6.4. That asymmetry runs against MLX: it is holding about 14% more language-model bytes than llama.cpp is. Keep that in mind when MLX wins anyway.

Benchmark protocol, identical for every llama.cpp row: llama-server with -ngl 99 -c 8192, three fixed prompts, n_predict: 256, temperature: 0, seed: 42, cache_prompt: false. Aggregate throughput is total predicted tokens over total predicted milliseconds, taken from the server's own timings object rather than a stopwatch. MLX ran the same three prompts at 256 tokens with a temperature-0 sampler.

Temperature 0 with a fixed seed matters more than it looks. It makes the token stream deterministic, which means the draft-acceptance counts are reproducible to the token, and I can compare acceptance across runs that differ wildly in wall-clock.

llama.cpp: every default MTP setting was slower

First the thing you get for free without asking, because it is a trap. Load an MTP GGUF normally and llama.cpp tells you exactly what it thinks of the head:

MTP is off by default in llama.cpp: fifteen tensors and 290.5 MiB are loaded then dropped, and enabling it cost 15% throughput at the recommended depth
Download a file with MTP in the name and change nothing, and you carry the head without using it. Ask for it and every draft depth was a regression.
text
1model has unused tensor blk.64.attn_q.weight (size = 43253760 bytes) -- ignoring
2model has unused tensor blk.64.ffn_down.weight (size = 73113600 bytes) -- ignoring
3model has unused tensor blk.64.nextn.eh_proj.weight (size = 36044800 bytes) -- ignoring
4...

Fifteen tensors, 304,601,088 bytes, 290.5 MiB, enumerated and then dropped on the floor. MTP is off by default. If you downloaded a file with MTP in the name and changed nothing else, you are carrying the head and not using it. You have to ask:

bash
1llama-server -m Qwopus3.6-27B-Coder-MTP-Q5_K_M.gguf -ngl 99 -c 8192 \
2  --spec-type draft-mtp --spec-draft-n-max 3

Do that and the log confirms it took:

text
1common_speculative_init_result: creating MTP draft context against the target model

So I asked, at every draft depth from 1 to 4:

config

tokens/sec

vs baseline

draft acceptance

no MTP

23.08

1.00x

—

--spec-draft-n-max 1

22.34

0.97x

79.3% (338/426)

--spec-draft-n-max 2

21.45

0.93x

69.0% (442/641)

--spec-draft-n-max 3

19.60

0.85x

59.2% (488/824)

--spec-draft-n-max 4

17.15

0.74x

49.2% (505/1027)

Every single one is a regression, and it gets monotonically worse the more you draft. At the depth the merge PR recommends, I lost 15% of my throughput.

The acceptance column is not the problem. 59.2% at depth 3 is in the same neighbourhood as the 72% the PR reports, and the shape is exactly right: the deeper you draft off a single-layer head running autoregressively, the more it drifts, so acceptance falls from 79.3% to 49.2%. The head works. The head is doing its job. The economics around it are what fail.

Where the time goes: ~30 ms per drafted token

Speculative decoding is a bet with a fixed structure. Each step you pay for n draft generations plus one verification pass, and in exchange you get somewhere between 1 and n+1 tokens. It wins when drafting is cheap relative to a decode step. So I worked out what a step actually cost.

With temperature 0 the arithmetic is clean. Every step yields one guaranteed token plus however many drafts were accepted, so the number of steps is just tokens - accepted:

config

tokens

accepted

steps

ms/step

tokens/step

no MTP

768

—

768

43.3

1.00

n-max 1

768

338

430

80.0

1.79

n-max 2

768

442

326

109.8

2.36

n-max 3

768

488

280

140.0

2.74

n-max 4

768

505

263

170.2

2.92

Now take the differences down that ms/step column:

text
1depth 1 -> 2 : +29.9 ms
2depth 2 -> 3 : +30.1 ms
3depth 3 -> 4 : +30.3 ms

Every additional drafted token costs a flat 30 ms. Three measurements, spread of 0.4 ms. That is about as clean a linear cost as you will ever measure on a real machine, and it is the entire article in one number.

Compare it to the thing it is supposed to be cheaper than. A full decode step through all 64 layers of a 27B model takes 43.3 ms. Running one extra layer to draft one token costs 30 ms, which is 69% of a full forward pass.

That sets the bar. Every draft costs 30 ms whether it survives or not, and each one that survives saves a 43.3 ms decode step, so drafting only pays when acceptance clears 30 / 43.3, which is 69%. Look back at the acceptance column with that number in hand. Depth 3 runs at 59.2% and depth 4 at 49.2%, both well under the bar, which is exactly why they lose 15% and 26%. Depth 1 runs at 79.3%, comfortably over the bar, and still loses 3%. The reason is in the step times: a depth-1 step should cost 43.3 + 30 = 73.3 ms and it measures 80.0. There is about 7 ms per step that is neither the draft nor a plain decode, and it is enough to eat a margin that was only ever 10 points wide.

Why is one layer so expensive? Not bandwidth, as far as I can account for it. A draft step needs the 65th layer (290.5 MiB) and the output projection over the 248,320-token vocabulary (output.weight, 1.04 GB, which the head shares rather than duplicating). Call it 1.35 GB. The main model moves about 18.3 GB in 43.3 ms, so it is running at roughly 420 GB/s. At that rate 1.35 GB should take 3.2 ms. It takes 30.

Warning

That last paragraph is arithmetic, not instrumentation. I did not profile inside llama.cpp to see where the 27 ms difference goes, and I am not going to assert a cause I did not measure. What I can say is that the draft step is nine times less efficient per byte than the main model's own forward pass, and that its cost does not scale with anything I varied. Fixed per-call overhead fits that shape. So would a draft path that quietly does more work than the head alone. I do not know which, and the log line "creating MTP draft context against the target model" is suggestive without being conclusive.

The one setting that beats baseline

If the problem is paying 30 ms for drafts that get rejected, the fix is to stop drafting when the head is not confident. llama.cpp has exactly that knob and it defaults to off: --spec-draft-p-min, which stops the draft loop once token probability drops below a threshold.

config

tokens/sec

vs baseline

acceptance

no MTP

23.08

1.00x

—

-n-max 3 --p-min 0.8

24.02

1.04x

94.9% (353/372)

-n-max 2 --p-min 0.9

23.74

1.03x

97.5% (275/282)

-n-max 3 --p-min 0.9

23.65

1.02x

96.9% (286/295)

-n-max 6 --p-min 0.9

23.10

1.00x

95.8% (298/311)

-n-max 4 --p-min 0.95

21.92

0.95x

95.8% (251/262)

So it can be rescued. Gate the drafting hard and acceptance jumps into the mid-90s, wasted draft calls mostly disappear, and the best configuration I found finally clears baseline:

bash
1llama-server -m Qwopus3.6-27B-Coder-MTP-Q5_K_M.gguf -ngl 99 -c 8192 \
2  --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.8

24.02 t/s against 23.08. A 4% gain, out of a feature the merge PR measures at 85% on CUDA.

And it is genuinely delicate. Push to --p-min 0.95 at depth 4 and you are back to a 5% loss despite 95.8% acceptance: the gate fires so aggressively that only 262 drafts survive across 517 steps, and the fixed cost of running the machinery outweighs the handful of tokens it buys. There is no broad plateau of good settings here. There is a narrow ridge, and everything either side of it is worse than not turning the feature on.

If you drive this through LM Studio rather than llama.cpp directly, the same controls are there under different names:

bash
1lms load <model> --speculative-draft-mtp \
2  --speculative-draft-max-tokens 3 \
3  --speculative-draft-min-continue-probability 0.8

--speculative-draft-min-continue-probability is the one that matters. Without it you are on the regression table, not the rescue table.

And then mlx-lm, which cannot do any of this, wins

Same model, same three prompts, same 256 tokens, no MTP head in the file at all:

text
1tg 256 tok in 9.56s = 26.79 t/s
2tg 256 tok in 9.40s = 27.22 t/s
3tg 256 tok in 9.43s = 27.16 t/s
4AGG mlx-lm: 768 tok / 28.4s = 27.06 t/s
5peak memory: 22.15 GB

Put the whole board together:

runtime

configuration

tokens/sec

mlx-lm 0.31.3

6-bit, no MTP available

27.06

llama.cpp b10330

Q5_K_M, MTP tuned (n 3, p-min 0.8)

24.02

llama.cpp b10330

Q5_K_M, no MTP

23.08

llama.cpp b10330

Q5_K_M, MTP at recommended depth 3

19.60

MLX is 13% faster than the best MTP configuration I could find, 17% faster than llama.cpp's plain baseline, and 38% faster than llama.cpp running MTP the way the PR suggests. It does this while holding about 14% more language-model bytes, because its quantisation is coarser. Correct for that and the gap widens.

That is the finding I did not go looking for. The interesting difference between these two runtimes on Apple Silicon right now is not the feature one of them has. It is the per-token efficiency of the plain decode loop, and MLX is simply better at it on this hardware. A 4% speculative-decoding win is noise next to a 17% baseline gap.

I wrote up separately whether two new models, Ricursive's Ripple-65B Preview and Liquid AI's LFM3-1.7B, run in llama.cpp and mlx-lm on a Mac: Ripple-65B Preview benchmark, plus LFM3-1.7B.

One incidental thing worth knowing if you are sizing context for this model family: only 16 of the 64 layers use ordinary KV attention. The rest are linear-attention layers carrying a recurrent state.

text
1llama_kv_cache: size = 256.00 MiB (4096 cells, 16 layers, 1/1 seqs)
2llama_memory_recurrent: size = 149.62 MiB (1 cells, 64 layers, 1 seqs)

256 MiB of KV for 4k context on a 27B model. Long context is cheap here in a way it is not on a conventional transformer.

MTP on MLX in October 2026: oMLX, MTPLX and Ollama run the head

Upstream mlx-lm deletes the head, but it isn't the only way to run MLX. oMLX added native MTP for Qwen3.5 and 3.6 in a 6 May dev build and shipped it in v0.3.9 on 21 May, and its oQ quantiser keeps the mtp.* weights in output folders with an -mtp suffix. MTPLX is built around MTP heads on Apple Silicon and ships Qwen3.6-27B MLX builds with the head in a separate mtp/weights.safetensors file. Ollama v0.32.6, on 4 August, says "the MLX engine now uses the model's MTP head for speculative decoding automatically" for Qwen3.5. We run MTP on MLX ourselves now, in our own engine, for the Qwen3.8-27B fine-tune benchmark.

I haven't measured any of them on this M3 Ultra, so I can't say whether they beat the 27.06 t/s plain MLX number above. Their published numbers come from other Macs: MTPLX reports 2.69x over its own plain decode for its Qwen3.6 27B Optimized Speed build on an M5 Max, and the head in that build is a calibrated INT4 copy of the original. On the llama.cpp side, someone saw the same thing before I did. Issue #23752, filed in May on an M1 Max with Qwen3.5-9B, called MTP on Metal a "net loss at every configuration". It was closed as "not a bug", with the PR author saying other users, Georgi included, had reported a sizeable speed-up on Macs.

The run where MTP looked like a 1.20x win

I nearly published the opposite of this article, so this section is the important one.

My first pass ran with LM Studio in the background holding a 27B model resident, 38 to 44 GB of it. Its own status field claimed the model was generating, but the process sat at ~5% CPU with no server log written in hours, so I read it as a session someone had walked away from and left the weights loaded. It looked harmless either way. Here is what I measured:

config

contended

quiet

no MTP

11.68 t/s

23.08 t/s

--spec-draft-n-max 3

14.06 t/s

19.60 t/s

verdict

MTP wins, 1.20x

MTP loses, 0.85x

Same binary, same model, same prompts, same seed. The contention did not just add noise, it inverted the sign of the result. The baseline degraded further than the MTP path did, so MTP came out ahead, and I would have written up a 1.20x speedup that does not exist.

The draft acceptance rate was 59.2% in both columns. Identical, 488 accepted of 824 drafted, to the token. Ratios of a deterministic token stream survive contention perfectly. Timings do not. If I had only reported acceptance I would have looked rigorous and been wrong.

What caught it was a bandwidth check rather than suspicion. An M=1 bf16 GEMV on this machine should come close to its ~819 GB/s spec:

python
1import mlx.core as mx, time
2
3N = 8192
4W = mx.random.normal((N, N)).astype(mx.bfloat16); mx.eval(W)
5x = mx.random.normal((1, N)).astype(mx.bfloat16); mx.eval(x)
6
7def gemv(reps):
8    ys = [x @ W for _ in range(reps)]      # independent, so they pipeline
9    mx.eval(ys); mx.synchronize()
10
11gemv(5)                                     # warm up
12best = float("inf")
13for _ in range(4):
14    t0 = time.perf_counter(); gemv(40); best = min(best, (time.perf_counter() - t0) / 40)
15
16print(f"{N * N * 2 / best / 1e9:.0f} GB/s")

Contended it read 540 GB/s. Quiet it read 667 GB/s. That gap was the tell.

Two smaller traps from the same afternoon, both of which cost me a run:

Chaining the matmuls as y = y @ W instead of collecting independent results measured 133 GB/s, against 540 GB/s for the independent form on the same machine in the same state. Four times too low, and nothing to do with contention: dependent operations serialise and you end up timing kernel launch latency rather than memory. Separately, running llama-cli with -v to capture metadata dropped generation from 13.1 to 11.0 t/s, which is why the verbose logs quoted in this article come from different runs than the timings do.

The discipline that survives all this is dull and worth stating plainly. Measure the machine before you measure the software, run the baseline again at the end of the session (mine closed at 23.06 t/s against an opening 23.08, so nothing drifted), and treat any result that agrees with the marketing as the one most in need of a second look.

What to actually do

  1. 1

    Check whether your GGUF even has the head

    run llama-server once and grep the log for nextn. If you see model has unused tensor blk.NN.nextn.* -- ignoring, the head is in your file and switched off.

  2. 2

    Measure your baseline first, on a quiet machine

    close other model runners, confirm nothing is resident, and record plain decode throughput with a fixed seed at temperature 0. Without this number nothing else you measure means anything.

  3. 3

    Do not enable MTP with defaults

    --spec-type draft-mtp --spec-draft-n-max 3 on its own cost me 15% on Apple Silicon. If you turn it on, turn on the gate with it.

  4. 4

    Start at --spec-draft-n-max 3 --spec-draft-p-min 0.8

    the only region I found that beats baseline, and it beats it by 4%. In LM Studio the equivalent is --speculative-draft-mtp plus --speculative-draft-min-continue-probability 0.8.

  5. 5

    Re-run the baseline afterwards

    if it no longer matches the opening number, throw the session away rather than the hypothesis.

  6. 6

    On Apple Silicon, try MLX before you tune any of this

    for this model family it was faster with no speculative decoding at all than llama.cpp was with the feature tuned.

What I did not test

The honest boundaries of the above, because the comparison has real limits.

This is one model family on one machine. Qwen3.6 is a hybrid linear-attention architecture with only 16 conventional KV layers out of 64, and I would not assume any of these ratios transfer to a dense transformer or to an MoE. The merge PR's 1.85x was measured on CUDA, where the balance between a draft call and a decode step is completely different, and nothing here contradicts it on that hardware.

The two quantisations are not matched. Q5_K_M against MLX 6-bit affine is close enough to compare directions but not close enough to quote a precise ratio, and I did not build a matched pair because doing it properly means converting both from bf16 source.

Three prompts at 256 tokens is a short-generation, short-context workload. Acceptance rates tend to be higher on repetitive or structured output, so a long code-generation run may treat MTP more kindly than my mix did. I also did not test batched or concurrent serving, where the verification pass amortises across requests and the arithmetic changes completely.

And I did not test PR #990, or any of the MLX runtimes that already run the head. If it merges, upstream mlx-lm gets native MTP, and builds converted the usual way, like the three on my disk, will need reconverting with the head kept.

The thing I would most like to be wrong about is the 30 ms. If someone profiles that draft path on Metal and finds it is fixed overhead rather than real work, this feature goes from a 4% curiosity to the speedup it was supposed to be, on every Mac running llama.cpp. Has anyone put an instrument on it?

If it is memory rather than throughput you are sizing for, I have since measured that on this same machine — Metal hands the GPU 77.76 GiB of the 96 GB installed, not the 75% everyone repeats, and the cache arithmetic has changed completely. That is in 96 GB is 77.76 GiB: the real memory ceiling on an M3 Ultra.

---

I run LeanZero, where a lot of the work is checking whether the thing in the release notes survives contact with real hardware. Everything above was measured on a Mac Studio M3 Ultra on 2026-08-10, with the exact versions named in the rig section. If you reproduce it and get different numbers, I would rather hear it than not.

ShareXLinkedIn

Comments

0

Add a comment

Confirm your address using the email we send. You can unsubscribe at any time. Your email stays private and this does not sign you up for the newsletter.