MXFP8 vs Q8: 10x the weight error, 1% the perplexity
Mihai Perdum
Author
14 min readAugust 1, 2026
Key takeaways
At the same 8.25 bits per weight, MXFP8 reconstructed real Qwen3-Coder-30B weights with 6.85% mean layer-output error against 0.67% for 8-bit affine — about 10x worse, on all nine tensor kinds.
End to end that 10x collapses to +1.05% perplexity. Weight-reconstruction error is a real measurement and a bad predictor of model quality; I nearly published the first number on its own.
The tiebreaker is speed, not accuracy: at identical footprint, affine decoded at 54.5 tok/s against MXFP8's 49.3 — about 10% faster for 1% better perplexity.
The W8A8 path that justifies MXFP8 runs for exactly one shape on this hardware — M=1. Every other size raises 'NYI for the general case'.
MXFP8 is a compute format for GPUs with native FP8 tensor cores. On Apple Silicon you pay its costs and collect none of its benefit — but the cost is far smaller than a weight-error table implies.
A page on this site has been ranking for mxfp8 vs q8 for months and it never once mentions Q8. That is my fault, so here is the comparison, run properly: the same weights, both formats, on one machine, with the numbers I actually got.
The short version is that MXFP8 lost, but the margin depends entirely on which number you look at, and I nearly published the wrong one. At identical storage cost it reconstructs real Qwen3-Coder weights about ten times less accurately than plain 8-bit affine. Run a whole model end to end, though, and that ten-times gap turns into one percent of extra perplexity. Both measurements are mine and both are correct; the distance between them is the most useful thing in this article, and it is a standing warning about proxy metrics.
The practical answer still comes out the same way — use affine — but for a much less dramatic reason than the weight table suggests, and it ends up being a speed argument rather than an accuracy one.
What the two things actually are
"Q8" means two different layouts depending on which world you came from, and the difference matters enough that I am going to pin both down before comparing anything.
In GGUF, Q8_0 is defined in llama.cpp's ggml-common.h:
Thirty-two int8 weights, one fp16 scale, no zero point — symmetric. That is 34 bytes per 32 weights, or 8.5 bits per weight.
In MLX, the equivalent is mode="affine" at 8 bits, which is asymmetric: it stores a scale and a bias per group, so it can pin both ends of the group's range rather than assuming symmetry around zero. Do not take my word for what the library supports — the installed version will tell you, and it is the only source that cannot be out of date:
python
1import mlx.core as mx
2print(mx.quantize.__doc__)
On mlx 0.32.0 that prints:
mode
group size
bits
scale type
bias
affine
32, 64, 128
2, 3, 4, 5, 6, 8
same as input
yes
mxfp4
32
4
e8m0
no
mxfp8
32
8
e8m0
no
nvfp4
16
4
e4m3
no
5 rows × 5 columnsHeader row enabled
MXFP8 is one of the OCP microscaling formats. Each element is an E4M3 float — one sign bit, four exponent bits, three mantissa bits — and each block of 32 shares one E8M0 scale, which is a raw power-of-two exponent with no mantissa at all. The element and scale types are pinned in MLX's own source rather than only in the docs — backend/cuda/quantized/qmm/qmm_utils.h maps mxfp8 to cutlass::float_e4m3_t elements with a cutlass::float_ue8m0_t scale, and the Metal path I am actually running selects the same scale type in backend/metal/kernels/fp_quantized.h (using ScaleType = metal::conditional_t<use_mx_scale, fp8_e8m0, fp8_e4m3>).
So the metadata budgets come out like this, and I measured them rather than deriving them, by quantising a 4096×4096 bfloat16 tensor and adding up the bytes of every array that comes back:
format
block
element
scale
bias
bits/weight
GGUF Q8_0
32
int8
fp16
none
8.500
MLX affine 8-bit, group 32
32
uint8
bf16
bf16
9.000
MLX affine 8-bit, group 64
64
uint8
bf16
bf16
8.500
MLX affine 8-bit, group 128
128
uint8
bf16
bf16
8.250
MLX mxfp8
32
E4M3
E8M0
none
8.250
6 rows × 6 columnsHeader row enabled
The first surprise is that MXFP8 is the smaller format. One byte of scale per 32 weights and no bias at all beats a 4-byte scale-and-bias pair per 64 weights. MXFP8 at 8.25 bits per weight is exactly the same size as affine at group 128, and smaller than GGUF's Q8_0.
That is the comparison worth running, so that is the one I ran: affine group 128 against mxfp8, both at 8.25 bits per weight. Anything else is comparing a format against a bigger one and calling the bigger one better.
The measurement
I pulled real weights. Not a synthetic tensor — a Gaussian random matrix has no outliers, and outlier structure is exactly the property that decides between a block-float format and an affine one, so measuring on one would have told me nothing.
You do not need a whole model for this. Qwen/Qwen3-Coder-30B-A3B-Instruct is 61 GB across 16 shards, and one shard is 4 GB and contains the embeddings, the attention projections, the MoE expert tensors and the routers of the early layers:
For each tensor I quantised, dequantised, and measured two different errors, because they are not the same number and the second is the one that matters:
python
1q = mx.quantize(w, group_size=gs, bits=8, mode=mode)2dq = mx.dequantize(*q, group_size=gs, bits=8, mode=mode)34err_w = mx.linalg.norm(dq - w)/ mx.linalg.norm(w)# how far the weights moved5y, yq = mx.matmul(x, w.T), mx.matmul(x, dq.T)6err_y = mx.linalg.norm(yq - y)/ mx.linalg.norm(y)# how far the OUTPUT moved
Twenty-four tensors, mean relative L2 error on the layer output:
tensor kind
affine g64
affine g128
mxfp8 g32
embed
0.543%
0.603%
6.112%
attn.q
0.620%
0.714%
7.086%
attn.k
0.599%
0.684%
7.145%
attn.v
0.602%
0.687%
6.982%
attn.o
0.548%
0.610%
6.591%
moe.gate
0.564%
0.634%
6.653%
moe.up
0.554%
0.617%
6.600%
moe.down
0.569%
0.637%
6.457%
router
0.709%
0.868%
7.842%
overall
0.589%
0.670%
6.847%
11 rows × 4 columnsHeader row enabled
At 8.25 bits per weight in both cases, that is 10.2x more error for MXFP8. There is no tensor kind where it wins, or ties, or comes close. The gap is smallest on the embeddings and widest on the routers, which is the worst possible place to be wrong in a mixture-of-experts model — a router error does not degrade an answer, it sends the token to a different expert.
Hold that number loosely, though. It is a weight-reconstruction error, and I show further down that it overstates the damage by roughly an order of magnitude once a real model runs.
Why it loses
Three mantissa bits. That is the whole story.
E4M3 gives each element eight steps per binade, so between 1.0 and 2.0 the representable values are 1.0, 1.125, 1.25 and so on up to 1.875 — which is what a round-trip through the format confirms:
The shared scale cannot rescue this. E8M0 is a bare exponent, so it can only shift a block into a different binade — it can never subdivide one. Every weight in the block lands on roughly three significant bits no matter what the scale does. Affine int8, by contrast, spreads 256 evenly-spaced levels across whatever range the group actually occupies, and because it carries a bias as well as a scale it can pin both ends of that range.
You can watch that difference directly. I built blocks of 32 with a known maximum and checked what came back:
block maximum
mxfp8 returns
affine int8 returns
1.0
0.875
1.0
1.5
1.5
1.5
1.875
1.75
1.875
2.0
1.75
2.0
4.0
3.5
4.0
100.0
96.0
100.0
1000.0
896.0
1000.0
8 rows × 3 columnsHeader row enabled
Affine reproduces the block maximum exactly, every time, by construction. MXFP8 frequently does not, and the pattern is hard to miss — whenever the maximum sits on a power of two it comes back 12.5% low. Across a whole 2048×2048 tensor the block maximum moves by 11.18% on average and up to 29.29% under mxfp8, against 0.000% under affine at any group size.
I want to be careful about what I am claiming there. I measured the behaviour; I did not manage to reverse-engineer MLX's scale-selection rule well enough to tell you whether it matches the OCP reference formula, and I am not going to guess at a bug I cannot demonstrate. The observable fact is that a power-of-two shared scale plus a three-bit mantissa does not pin the top of a block, and an affine scale-and-bias pair does.
The case for MXFP8, and why it did not show up
Block-float formats are supposed to earn their keep on outliers. A single huge weight in a block drags an affine scale so wide that every other weight in that block collapses into the same few levels, whereas a shared exponent rides it out. That is a real effect and I could reproduce it — here is a block of thirty-one 1.0s plus one outlier, showing what happens to the 1.0s:
outlier
mxfp8 keeps 1.0 as
affine g32 keeps 1.0 as
16
1.000000
1.000000
256
1.000000
1.000000
4096
1.000000
−0.000242
65536
1.000000
−0.003876
5 rows × 3 columnsHeader row enabled
At a 4096x outlier the affine block is destroyed and MXFP8 is untouched. So the mechanism is genuine. It just never fires on these weights: across the Qwen3-Coder tensors I measured, the median in-block dynamic range was 39–51 dB, or a ratio of roughly 90:1 to 350:1 between the largest and smallest non-zero weight in a block of 32. Int8's 256 levels swallow that comfortably. On my own numbers affine was still exact at a 256:1 in-block ratio and destroyed at 4096:1, so the breaking point sits somewhere in between — well clear of anything these tensors do. Trained transformer weights, at least these ones, are simply not that spiky at 32-element granularity.
There is a neat piece of corroboration for this sitting in public. The community MXFP8 build of Qwen3-Coder-Next, mlx-community/Qwen3-Coder-Next-mxfp8, has a base config of {"group_size": 32, "bits": 8, "mode": "mxfp8"} — and then 96 per-layer overrides, one for every mlp.gate and mlp.shared_expert_gate, each reading {"group_size": 64, "bits": 8} with no mode field. MXFP8 does not support group 64. So what mode do those layers actually load as? mlx-lm passes the override dict straight to to_quantized(), whose mode argument defaults to "affine", which I checked rather than assumed:
The shipped MXFP8 model quietly keeps all 96 of its router layers on affine int8. That is the exact tensor kind where my measurements put MXFP8 at its worst.
It is not faster either
An accuracy loss buys something, normally. So I measured throughput on the same box — a Mac Studio, M3 Ultra, 60 GPU cores, 96 GB.
One trap first, because it cost me an hour. A single 4096² quantised matmul inside its own mx.eval() measures about 300 µs of Python and graph-building overhead and almost none of the kernel. Chain forty of them into one eval, mx.synchronize() before stopping the clock, and take the best of several runs. The same code went from 1559 µs to 41.5 µs. A bf16 matvec that reports 808 GB/s against this machine's 819 GB/s spec is a measurement you can trust; one that reports 108 GB/s is measuring your harness — I got that number the first time, because a model download was running in the background.
Decode shape, M=1, which is memory bound:
kernel
time
effective
vs bf16
bf16 dense
41.5 µs
808.2 GB/s
1.00x
affine g64, W8A16
23.8 µs
750.2 GB/s
1.75x
affine g128, W8A16
22.2 µs
779.9 GB/s
1.87x
mxfp8 g32, W8A16
27.5 µs
628.7 GB/s
1.51x
mxfp8 g32, W8A8 (qqmm)
73.1 µs
236.8 GB/s
0.57x
6 rows × 4 columnsHeader row enabled
Prefill shape, M=512, which is compute bound:
kernel
time
throughput
vs bf16
bf16 dense
1034.7 µs
16.60 TFLOP/s
1.00x
affine g64
1104.3 µs
15.56 TFLOP/s
0.94x
affine g128
1084.9 µs
15.84 TFLOP/s
0.95x
mxfp8 g32
1097.5 µs
15.65 TFLOP/s
0.94x
5 rows × 4 columnsHeader row enabled
Two things fall out. At decode, both formats beat dense bf16 because they move half the bytes, but affine converts that into more speed than MXFP8 does — it sustains 780 GB/s against MXFP8's 629, so MXFP8 gives up about a fifth of the bandwidth advantage its smaller footprint should have earned. And at prefill, every quantised mode is slower than dense bf16. Quantisation on this machine is a memory optimisation, not a compute one, and if your workload is prefill-heavy it is costing you.
Then there is the W8A8 row, which is the real point of MXFP8's existence. Quantising the activations as well as the weights is where a block-float format pays off, and MLX exposes it — nn.quantize takes a quantize_input=True flag, and it is supported for exactly two modes:
python
1if mode notin["nvfp4","mxfp8"]:2raise ValueError(3f"Quantized activations are only supported for 'nvfp4' and 'mxfp8' modes, got {mode}."4)
Affine cannot do this at all. This is the one genuine architectural advantage MXFP8 has. So I swept mx.qqmm across fourteen batch sizes from 1 to 256:
text
1M=1 OK
2M=2 [QQMatmul] NYI for the general case
3M=4 [QQMatmul] NYI for the general case
4M=8 [QQMatmul] NYI for the general case
5...
6M=256 [QQMatmul] NYI for the general case
One shape out of fourteen. And at that one shape it ran at 0.57x of dense bf16 — slower than doing no quantisation at all.
So why does this format exist
Because it was not designed for this machine. MXFP8 is a compute format, aimed at hardware with native FP8 tensor cores, where the win is that the multiply-accumulate itself runs in FP8 and you never materialise a wide intermediate. On an NVIDIA B200, PyTorch and TorchAO report end-to-end speedups of up to 1.26x for MXFP8 and 1.68x for NVFP4 — their numbers, not mine, on hardware I do not have.
Apple Silicon has no equivalent path today. Apple's own M5 write-up introduces GPU "Neural Accelerators" with dedicated matrix-multiplication units and reports up to 4x time-to-first-token against an M4. It does not say in so many words that M3 lacks them, so I will not claim that it does — I will only say what I measured on this M3 Ultra, which is that the W8A8 kernel is unimplemented for every shape but one, and that where the weight-only kernel does exist it is slower than the affine one.
That is the whole asymmetry. On Blackwell, MXFP8's accuracy cost buys a real throughput win. Here it buys nothing at all — which brings me to how large that cost actually is, because up to this point I have only shown you what happened to the weights.
Warning
None of this transfers to MXFP4 or NVFP4 by analogy. Those are 4-bit formats competing against 4-bit affine quantisation, where the trade-offs are completely different and block-float formats do much better. I measured 8-bit against 8-bit. Do not generalise it a bit-width sideways.
The number that nearly fooled me
Everything above is a proxy. Weight error and layer-output error are real measurements, but nobody runs a norm — they run a model. So I quantised an entire model three ways and measured what a user actually gets.
Qwen/Qwen3-4B, converted with mlx_lm.convert, evaluated on ~16k tokens of held-out TypeScript and TSX the model had never seen (this is a coding question, so the eval text is code, not wikitext), plus a decode-throughput run on each:
variant
bits/weight
disk
peak RAM
perplexity
decode
affine g64
8.500
4.27 GB
6.71 GB
2.4627
51.0 tok/s
affine g128
8.250
4.15 GB
6.58 GB
2.4633
54.5 tok/s
mxfp8 g32
8.250
4.15 GB
6.58 GB
2.4886
49.3 tok/s
4 rows × 6 columnsHeader row enabled
MXFP8 costs +1.05% perplexity against affine g64, and +1.03% against affine g128 at the identical footprint. Not ten times anything. A tenfold gap in weight-reconstruction error came out the far end as a one-percent difference in how well the model predicts code.
That is worth sitting with, because I had a finished draft that led with "ten times the error" and it would have been technically true and practically misleading. Transformers are enormously tolerant of weight perturbation — the error is spread quasi-randomly across a very wide network and most of it cancels rather than compounds. A relative-L2 table tells you what happened to the weights. It does not tell you what happens to the model, and the ratio between those two things is not fixed.
So the real case against MXFP8 here is not accuracy, it is that it loses on both axes at once. At an identical 4.15 GB and identical peak memory, affine g128 gives you slightly better perplexity and decodes about 10% faster. There is no trade to make. If MXFP8 had been meaningfully faster, a 1% perplexity cost would have been an easy price.
Note
The two halves of this article use two different models, deliberately. Weight-level error is measured on real Qwen3-Coder-30B-A3B-Instruct tensors, because that is the model people actually ask about. The end-to-end run is Qwen3-4B, because quantising a model three ways needs all of it, and 61 GB was not going to finish over an unauthenticated Hugging Face connection. A 4B dense model is not a 30B MoE, and the MoE routers are exactly where MXFP8 measured worst — so if anything I would expect the end-to-end penalty on the coder to be a little larger, not smaller.
What to actually do
If you are running a coding model locally on Apple Silicon and you want 8-bit weights, use affine.
bash
1# -q is required — without it the other flags are ignored and nothing is quantised2mlx_lm.convert --hf-path Qwen/Qwen3-Coder-30B-A3B-Instruct \3-q --q-mode affine --q-bits 8 --q-group-size 128 --mlx-path ./out
Group 128 is the one I would actually ship: it matches MXFP8's footprint exactly, edges it on perplexity, and gave me the fastest decode kernel of anything I measured. Group 64 buys a marginally better weight reconstruction for an extra 0.25 bits per weight and decoded slower, which on this evidence is not a trade worth making. In GGUF land Q8_0 is the same family of idea and lands at the same 8.5 bits per weight as affine g64.
Reach for MXFP8 when you are on hardware with native FP8 matmul and you are quantising activations too. That is the workload it was built for, and none of my criticism here touches it. On a Mac there is currently no configuration where it wins: same size, slightly worse perplexity, slower decode.
What I did not test
I did not test MXFP8 on hardware that supports it natively, so all of this is a statement about Apple Silicon and not about the format in general. I did not test 4-bit formats at all. And the perplexity run is one model on one corpus of one language — it establishes the order of magnitude of the penalty, not a precise constant.
Every number here came off one Mac Studio this week, and the scripts are short enough that you can reproduce the lot in an afternoon. If you have run the same comparison on a Blackwell card, I would genuinely like to see where the crossover sits — at what point does the compute win pay back a 10x weight error?
