v0.5 of our Atlassian Qwen3.8-27B: beats v0.4, now with GGUF for llama.cpp and Ollama
Mihai Perdum
Author
11 min readSeptember 20, 2026
Key takeaways
v0.5 of our Qwen3.8-27B fine-tune for Forge, Jira, Confluence and JSM beat v0.4 on every one of our four release gates, with no waiver needed — the first round that's happened for.
The gate itself got harder for this release: 25 briefs at a single pass became 35 briefs measured three times and averaged, with the bar raised to 30 of 35. v0.5 scored 30.0/35 apps and 31.3/35 manifests on that harder gate; v0.4 re-scored on the same gate lands at 22.7/35 and 30.7/35.
The training method changed, not just the round: v0.5 adds a KL-divergence penalty against two teacher models on top of the usual fine-tuning loss, specifically because a plain capacity increase (more layers, higher LoRA rank) kept buying app-writing ability while quietly losing the held-out factual-recall set that v0.4 protects.
GGUF quantized versions for llama.cpp and Ollama, Q8_0 and Q6_K, are published on Hugging Face alongside the MLX release we already ship. All three parity-proof links, merge fidelity, cross-runtime conversion fidelity, and per-quant KL divergence, plus the served-model task probe through llama-server, are in: the Q8_0 file matches the MLX release on identifiers (85% pre-April / 15% post) and lands within the gate's own noise on shape (28.7/35 apps against the MLX release's 30.0/35).
One real weakness, stated plainly rather than smoothed over: identifiers about anything after April 2026 keep getting worse as training capacity goes up, round over round. It's flagged, not fixed, and not part of the release gate.
v0.5 of the Qwen3.8-27B we fine-tune for Forge, Jira, Confluence and JSM is out, and it beat v0.4 on every gate we measure it against, no waiver needed this time. GGUF versions for llama.cpp and Ollama, Q8_0 and Q6_K, are now published alongside the MLX release we already ship — the parity numbers are good, and this post says exactly what was measured and how.
v0.5: Qwen3.8-27B beats v0.4 on every measured gate
We raised the bar for this round before we ran it. v0.4 shipped on a 25-brief gate, one pass, bar 20 of 20. v0.5 was measured on 35 briefs, three passes averaged, bar 30 of 30 — a genuinely harder test, not the same scale with a bigger number. Re-scored on that harder gate, v0.4 lands at 22.7 of 35 apps and 30.7 of 35 valid manifests. v0.5 scores 30.0 of 35 apps and 31.3 of 35 manifests, a margin of 7.3 apps against a rule that only requires 3.
The release rule has four conditions, and all four passed clean: the held-out factual-recall set didn't regress, identifiers held within tolerance, shape improved by more than the required margin, and looping stayed at or below the untouched base on every leg, short and long-context both. v0.4 needed one of those four waived on its own card, with the reasoning shown, to ship. v0.5 didn't need the waiver.
That milestone, no waiver, is worth pausing on, because the round before this one is exactly why it's not a given.
What changed in T9, the round behind v0.5
Internally this round is called T9, and it exists to fix a problem a prior, unshipped experiment exposed. We tried giving the adapter more capacity, more of the model's layers, a higher LoRA rank, the same recipe otherwise, and it worked exactly the way more capacity usually does: more apps written correctly, a better shape score. It also broke the one thing v0.4's release rule exists to protect. The held-out factual-recall set, the one measuring whether the model still knows real Atlassian identifiers rather than confident near-misses, regressed by 0.135 against a rule that allows 0.010. That round was never released; the rule did its job and kept the incumbent.
T9 trained a bit longer than the round it fixes, 2,600 steps against 2,400, on a slightly larger mix. Neither of those is the actual fix. The fix is a different loss: alongside the usual fine-tuning objective, a KL-divergence penalty pulls the model's predictions back toward two teacher models on the rows where accuracy matters most, the base model plus the incumbent release on knowledge-heavy rows, the untouched base itself on general-purpose rows. The point is letting capacity buy shape and app-writing ability without spending the model's grip on facts to do it — the same LoRA mechanics we walked through from scratch on the 9B sibling model, just with a second loss term added on top.
It worked better than we'd measured before: the held-out factual-recall loss didn't just avoid regressing, it improved past every prior round, including the ones with far less capacity. That's the first time a round has improved this particular guard rather than just staying inside it.
One honest weakness survives the improvement, and we're stating it rather than letting the headline numbers cover for it: identifiers about anything after April 2026 keep getting worse as capacity goes up, round over round. It isn't part of the release rule, and it isn't fixed by this round. It's a real, measured, ongoing weak spot, and we'd rather say so here than have someone find it first.
GGUF for llama.cpp and Ollama, alongside MLX: shipped, parity proven three ways
We publish the MLX release already, adapter, fused 8-, 6- and 4-bit quantized weights, and the untouched base, all on Hugging Face. GGUF for llama.cpp and Ollama is now published alongside it, two quant levels, Q8_0 and Q6_K. Building it well meant more than running one conversion script: there's no single tool that can compare a GGUF file against an MLX file directly, so proving the conversion is faithful took three separate measured links, each inside the one tool that can actually measure it.
The first two links are clean. The adapter, merged into the original bf16 checkpoint rather than the MLX-quantized base, reproduces the MLX release's own greedy output on 16 of 20 test prompts, and where it diverges, it diverges late, at genuine near-ties, not from the first token. Converting that merged file into a GGUF, a step that's supposed to be a lossless container change, checks out the same way against the merged bf16, across two completely different inference engines, mlx-lm and llama.cpp, on 17 of 20.
The third link is the one that actually measures what quantization costs, in KL divergence rather than a file-size ratio: how far each quant level's predicted-token distribution has drifted from the full-precision reference, averaged across 40 chunks of held-out validation text. Both quant levels we're shipping came back well inside the range the quantization community treats as safe for their respective bit widths:
text
1Q8_0 Mean KLD: 0.000842 ± 0.000276
2Q6_K Mean KLD: 0.003060 ± 0.000273
We're not shipping a 4-bit GGUF in this release. The command to build one is a single argument different from the ones we ran, and we deliberately didn't run it: a quant doesn't get a "matches the original" claim without a KLD number and a task-probe number published beside it, and we haven't measured that ladder for 4-bit yet.
The last link is the served-model task probe: the same identifier and app-writing checks we run on the MLX release, run against the GGUF through llama.cpp's own server, to confirm the model still does the thing it was tuned to do, not just that its logits are close. We ran that against the Q8_0 file specifically — it matches the MLX release on identifiers, 85% pre-April accuracy and 15% post-April, none of the 26 probe answers truncated, and lands within the gate's own noise on shape, 28.7 of 35 apps and 30.3 of 35 manifests (mean of three passes) against the MLX release's own 30.0 and 31.3 on the same briefs. We didn't run a separate served pass against Q6_K; its parity claim rests on its KLD number alone, the same standard we hold every quant to before shipping it, just without the extra served-probe step we happened to run on Q8_0. We're also writing up the exact method, every command, and the bugs that cost us rebuilt measurement steps along the way, as a separate tutorial for anyone converting their own fine-tune the same way.
Running the GGUF with llama.cpp and Ollama
Both quant levels run the way any GGUF does: llama-server or llama-cli directly against the file for llama.cpp, or ollama run hf.co/Mihai-LeanZero/<repo>:Q8_0 for Ollama, which pulls straight from the Hugging Face repo, the same pattern the rest of the Qwen3.8-27B GGUF ecosystem already uses. One thing worth knowing before you reach for it: every other Qwen3.8-27B GGUF on Hugging Face right now is a general-purpose or "uncensored" re-quant of the base model. None of them know anything about Forge, Jira, Confluence or JSM specifically, because none of them are trained on it. This is the one that is.
Qwen3.8-27B and Qwen3.5-9B were taught Forge, Jira and Confluence on a single Mac Studio with 96 GB. This is the whole procedure with the seven scripts that do it, printed in full: a pinned MLX environment with two patches, an 8-bit base with its speculative-decoding head kept as a sidecar, a chat-JSONL dataset built from a folder of Markdown, a segmented trainer with full-state checkpoints, an adapter expander, a per-module merge into the 8-bit shards. The kit was run end to end for this page on the 9B, in 21 minutes, and its real output follows each step, including the one where the validation loss turns the wrong way.
