Qwen3.8, quantised by us from bf16 with its speculative-decoding head intact, then taught Forge, Jira, Confluence and Jira Service Management from real apps, the documentation, the OpenAPI specs and answers we gave on the Atlassian communities. Every number on this page was measured, and the ones that are still weak are printed anyway.
Qwen3.8-27B is a strong open model that knows almost nothing about Atlassian Forge: asked to write a Forge app, it invents a manifest format and imports packages that do not exist. We taught it. The result is the same model, in the same file format, that now writes Forge apps which pass Atlassian's own validator and compile, answers questions about Jira, Confluence and JSM with the real identifiers, and does it locally on a Mac.
Every claim on this page comes from a test we ran and can rerun. The table further down shows every round of training side by side, including the one we rejected and the untouched base, so you can see what improved, what did not, and by how much. If you only read one row, read "complete app passes every gate": that is a one-line brief turned into a whole app that Atlassian's validator accepts and TypeScript compiles.
To use it: download the 8-bit folder from Hugging Face, open it in goose local edition, LM Studio or mlx-lm, ask with thinking on for questions and thinking off for code, and keep the manifest validator and tsc in the loop, exactly as our harness does. It is a strong assistant, not a compiler.
One standalone MLX model folder per size. It is a normal Qwen3.8 checkpoint, so anything that loads an mlx-community model loads it. LoRA was only the training method; you never have to know what an adapter is unless you want one.
| Artefact | What it is | Size |
|---|---|---|
| Qwen3.8-27B-Atlassian-Q8-mlx | The model. 8-bit, MTP head kept, adapter merged in. Loads in goose, LM Studio, mlx-lm. | ~31 GB |
| Qwen3.8-27B-Atlassian-lora | The adapter alone (rank 32, ~190 MB) for mlx-lm's --adapter-path and Rapid-MLX's serve --adapter-path. Download it to a local folder first; mlx-lm resolves adapters from disk only. | 190 MB |
| Qwen3.8-27B-Atlassian-Q8-base-mlx | The untouched quantised base we trained on: the same weight class as lmstudio-community's 8-bit, rebuilt from bf16 with the MTP sidecar. | ~31 GB |
| Qwen3.8-27B-Atlassian-Q6-mlx | Plain 6-bit (group size 64) quantised from the merged bf16 weights. | ~21 GB |
| Qwen3.8-27B-Atlassian-Q4-mlx | Plain 4-bit (group size 64) from the merged bf16 weights. Retention of the tuning at 4 bits is measured only by agreement with the 8-bit model and by the identifier probe printed on its card; the app-writing tasks were not re-run on it. | ~15 GB |
| Qwen3.5-9B-Atlassian-Q8-mlx | The small one: the same recipe and data on Qwen3.5-9B, with its own adapter, base, Q6 and Q4 under the same repo names. Compared with its base and with the 27B further down. | ~10 GB |
Mount the folder in the MLX Engine view; goose's engine supervisor finds mtp.safetensors and turns speculative decoding on. Measured on the merged model, single stream: 1.26× faster decode on short context, 1.21× at 32k.
Learn moreDrop the folder into your models directory. It loads with the MLX engine like any mlx-community model.
mlx_lm.generate --model <folder>. Single-user servers can add the DFlash2 drafter for 2.08× decode on the 8-bit base.
Learn moreSame harness, same prompts, same machine. The untouched base is the quantised Qwen3.8-27B we trained on; “tuned” is release v0.3.
| Probe | Base | Tuned |
|---|---|---|
| Atlassian identifiers, facts before 2026-04 (13 questions, thinking on) | 15% | 69% |
| Atlassian identifiers, facts after 2026-04 (13 questions) | 23% | 23% |
| Complete Forge app from a one-line brief: manifest passes Atlassian's validator (tuned: thinking off; base: its best mode) | 0 / 25 | 14 / 25 |
| …and the whole app also compiles under tsc against the real @forge/* types | 0 / 25 | 12 / 25 |
| Genuine loops, 40 prompts × 4 sampling legs at 2048 tokens | 2 / 1 / 3 / 4 | 0 / 0 / 1 / 0 |
| Answers that never terminated, same four legs | 60 / 48 / 53 / 25% | 25 / 15 / 28 / 13% |
| Needle recall | 100% at 4k, 100% at 32k, 100% at 128k | 100% at 4k, 100% at 32k, 100% at 128k |
| Genuine loops on 32k and 128k context | 1 (at 128k) | 0 |
| MTP draft acceptance, same benchmark | 53% | 51% |
forge deploy. With thinking off the usual one is a TypeScript typing error against @forge/react or @forge/bridge; the next is a resolver or resources key placed under modules, or a handler path that does not match the file. Pair the model with the manifest validator and tsc, exactly as our harness does. Use thinking for questions and thinking off for code generation; we measured the difference and it is large.Same harness, same prompts, same machine. Each round starts from the previous winner's adapter. Release T4 is the column the models on Hugging Face were built from; n/a means the test was not run on that round (the rejected experiment was stopped after its first probes).
| test | untouched base | T1 | T2 (experiment, rejected) | T3 | T4 |
|---|---|---|---|---|---|
| trained from / mix / steps | the quantised base | base / m3 / 2500 | t1 / m4 / 400 | t1 / m6 / 1500 | t3 / m7 / 1500 |
| training hours, this round | 0 | 11.3 | n/a | 7.1 | 7.6 |
| validation loss, round-1 mix (general Atlassian), lower is better | n/a | 0.874 | n/a | 0.844 | 0.825 |
| validation loss, round-3 mix (Forge-heavy) | n/a | 0.686 | n/a | 0.632 | 0.596 |
| identifiers pre-2026-04 / post, thinking on | 15% / 23% | 77% / 31% | 62% / 31% | 69% / 23% | 69% / 23% |
| identifiers, thinking off, served | n/a | 62% / 15% | n/a | 85% / 31% | 62% / 31% |
| manifests valid, of 25 | 0 | 14 | n/a | 13 | 14 |
| complete app passes every gate, of 25 | 0 | 3 | n/a | 11 | 12 |
| genuine loops per leg (official / t0.6 / greedy / instruct) | 2 / 1 / 3 / 4 | 2 / 1 / 2 / 0 | 0 / 0 | 2 / 0 / 0 / 0 | 0 / 0 / 1 / 0 |
| non-termination per leg | 60 / 48 / 53 / 25% | 20 / 20 / 13 / 10% | 20 / 28% | 20 / 18 / 10 / 15% | 25 / 15 / 28 / 13% |
| long context, 32k / 128k | 32k: 0 loops, 4/4 cut off; 128k: 1 loops, 4/4 cut off | 32k: 0 loops, 1/4 cut off; 128k: 0 loops, 2/4 cut off | n/a | 32k: 0 loops, 1/4 cut off; 128k: 0 loops, 0/4 cut off | 32k: 0 loops, 2/4 cut off; 128k: 0 loops, 2/4 cut off |
| needle recall | 4k 100%, 32k 100%, 128k 100% | 4k 100%, 32k 100%, 128k 100% | n/a | 4k 100%, 32k 100%, 128k 100% | 4k 100%, 32k 100%, 128k 100% |
| leak probe: regex matches / memorised | 34 / 0 | n/a | n/a | 26 / 0 | 20 / 0 |
| merged 8-bit vs base+adapter: KLD / top-1 | not built | not built | not built | 0.0452 / 99.37% | 0.0407 / 99.43% |
| MTP acceptance / decode speedup at 128, 2k, 8k, 32k | 53% / 1.55×, 1.53×, 1.23×, 1.22× | n/a | n/a | 53% / 1.3×, 1.34×, 1.36×, 1.22× | 51% / 1.26×, 1.38×, 1.34×, 1.21× |
| LM Studio load | n/a | n/a | n/a | PASS | PASS |
A rule in code decides. A round replaces the previous winner only if all four hold: no regression on the round-1 validation set beyond 0.010; identifiers with thinking on within one question; at least as many complete apps passing every gate, with manifests within two; loops at or below the untouched base on every leg and non-termination not more than 5 points above it.
Sample counts per source in each round's mix, after exact de-duplication. Round 1 established the base of knowledge; round 3 added the documentation as reading material and the model's own gate-passing apps; round 4 added a second, harder rejection-sampling harvest and the model's own validator-corrected mistakes.
| source | T1 (m3) | T2 (experiment, rejected) (m4) | T3 (m6) | T4 (m7) |
|---|---|---|---|---|
| LeanZero Forge apps (files, configs, tests) | 580 | 580 | 580 | |
| schema-generated manifests + fix pairs | 2510 | 1200 | 2510 | 2510 |
| UI Kit examples that compile | 486 | 486 | 486 | 486 |
| documentation question-answer (verbatim-gated) | 4000 | 6000 | 6000 | |
| developer-community threads | 220 | 220 | 220 | |
| community-research compendium + posted answers | 1600 | 1200 | 1200 | |
| OpenAPI endpoint and scope facts | 2500 | 1500 | 1500 | |
| brief-to-app samples (gated) | 400 | 400 | 400 | 400 |
| general-code replay | 831 | 271 | 831 | 831 |
| @forge/* version facts | 96 | 96 | 96 | |
| structural fix pairs (synthetic) | 257 | 257 | 100 | |
| documentation sections as text | 4117 | 4117 | ||
| own gate-passing apps, round 1 | 431 | 431 | ||
| own gate-passing apps, round 2 (harder briefs) | 937 | |||
| own validator-corrected mistakes | 362 | |||
| tokens (approx.) / steps / hours | 3.7M / 2500 / 11.3 | 0.9M / 400 / n/a | 5.5M / 1500 / 7.1 | 6.1M / 1500 / 7.6 |
Rejection sampling, the lever that moved the compiling-app count: round 1 sampled 4 apps for each of 300 fresh briefs from T1 and 63% passed every gate; round 2 sampled 4 for each of 450 harder briefs from T3 and 72% passed. Only passers are trained on, plus "fix this manifest" pairs built from the failures whose sibling passed, with the validator's real messages as the prompt.
The 25 shape tasks, failures classified from the validator's and tsc's own messages. This is the list the next round trains against.
The harness, the gates and the thresholds, so the numbers above mean something.
Mean token cross-entropy of the adapter on a fixed 32-sample subset of a mix's held-out split (seed 7), the same subset for every round. m3 is the round-1 mix (general Atlassian knowledge), m6 the round-3 mix. A round that regresses on m3 by more than 0.010 is rejected; that is the test that eliminated the T2 experiment (0.892 vs 0.874).
26 short questions with a strict output contract, graded by a regex on the last line: 13 about identifiers that existed before April 2026 and 13 about the June 2026 @forge/* wave (package majors, module keys, tsc traps). One sample per question at temperature 1.0, so one question is 8 points and two adjacent rounds can differ by a question on noise alone. Run twice: thinking on through mlx-lm, thinking off through Rapid-MLX.
25 one-line briefs ('Write a JSM portal request-detail panel showing linked assets'), thinking off, served through Rapid-MLX with MTP. Each answer is unpacked into a project and graded by @forge/manifest's own validator, our allow-list of real module types and keys (the official validator accepts a misspelled key), and tsc against the pinned @forge/react 12.1.3 and @forge/bridge 6.3.1 types. 'Manifest valid' = validator plus allow-list; 'full gate' = the whole app also type-checks. This is the deliverable metric.
40 prompts (20 Atlassian, 20 general) x 4 sampling legs at 2048 tokens: thinking with Qwen's official sampling (1.0/0.95/20), thinking at temperature 0.6, greedy, and instruct mode with no repetition penalty. A loop = a 40-character window repeated 8+ times or under 25% distinct 4-grams in the last 512 tokens. Non-termination = the answer was still going at the cap. Gate: loops must not exceed the untouched base on any leg and non-termination must not rise more than 5 points.
The same loop metrics on 4 answers each at 32k and 128k tokens of context (token-calibrated windows, not word counts), because looping shows at 128k long before it shows at 2k.
A random 8-digit key hidden at depths 0.15/0.35/0.5/0.85 in 4k, 32k and 128k token contexts; 4 trials per length; the model must return the key.
24 adversarial prompts x 3 samples asking for tenant hostnames, tokens, e-mails and client names, plus 40 forced-prefix completions, scanned with a regex for credential shapes and real tenant patterns. Every regex match is then checked against the training corpus: a match that exists there is a memorised leak; a match that does not is an invented shape. The untrained base is run as a control.
After the adapter is merged into the 8-bit weights per module, the merged model's top-1024 logits are compared with base-plus-adapter served by mlx-lm on 200 frozen prompts: mean KL divergence and top-1 agreement. Gate: KLD at most 1.5x the base's own distance from bf16 (0.0446) and top-1 at least 90%. The 6- and 4-bit family members are scored the same way plus the identifier probe.
Rapid-MLX serves the merged model with its multi-token-prediction sidecar off and on; decode tokens per second at 128, 2k, 8k and 32k tokens of prompt (median of 2 runs), draft acceptance from the engine's metrics, and greedy byte-equality on 20 short prompts.
The merged folder is linked into LM Studio's models directory, loaded with its MLX engine through the lms CLI, and asked one Forge question; the answer must name the right module.
Same recipe, same data as the 27B's last round, trained in one run of two full epochs, and put through the identical harness. It is compared with its own untouched base and with the 27B release. The compiling-app count is the number to look at: training budget on the right data matters more than parameter count for this task.
| test | Qwen3.5-9B untouched | Qwen3.5-9B tuned | 27B T4 |
|---|---|---|---|
| identifiers pre / post, thinking on | 15% / 15% | 62% / 15% | 69% / 23% |
| identifiers, thinking off, served | n/a | 69% / 38% | 62% / 31% |
| manifests valid, of 25 | n/a | 19 | 14 |
| complete app passes every gate, of 25 | n/a | 16 | 12 |
| genuine loops per leg | 0 / 0 / 5 / 3 | 0 / 0 / 4 / 0 | 0 / 0 / 1 / 0 |
| non-termination per leg | 28 / 23 / 33 / 23% | 18 / 8 / 13 / 0% | 25 / 15 / 28 / 13% |
| leak probe | - | LEAK PROBE PASS: 0 leaks over 112 outputs (26 regex matches: {'hallucinated': 21, 'hallucinated-token': 5}) | PASS, 0 memorised |
| merged 8-bit vs base+adapter: KLD / top-1 | - | 0.0537 / 99.14% | 0.0407 / 99.43% |
| MTP acceptance / decode speedup | - | 67% / 1.14×, 1.09×, 1.08×, 1.05× | 51% / 1.26×, 1.38×, 1.34×, 1.21× |
| LM Studio load | - | PASS | PASS |
The engineering is in the gates. Each one exists because an earlier attempt failed it.
Top-1024 KL divergence against the bf16 teacher on 200 frozen prompts. The 8-bit base scores 0.0446 with 99.35% top-1 agreement. An Unsloth-recipe 6-bit candidate built from the same source with their imatrix was scored too and fell under the 99% top-1 floor, so the base is plain 8-bit and the smaller family members are plain 6- and 4-bit quantisations.
Qwen3.8's multi-token-prediction head is shipped as a sidecar and re-benchmarked after the merge: draft acceptance is unchanged on the benchmark (53% before, 51% after) and runs around 67% on code-generation workloads, where the tuned model's output is more predictable.
Every generated app goes through @forge/manifest's own validator, our allow-list of real module types and keys (the official validator accepts a misspelled key), and tsc against the pinned @forge/react and @forge/bridge type definitions. A manifest that merely looks right does not count.
Forty prompts across four sampling regimes at 2048 tokens, plus a long-context leg at 32k and 128k. The tuned model must loop no more than the base on every leg. Non-termination fell on all four.
Training data is LeanZero-written Forge apps, public documentation, OpenAPI specs, our own community answers and 220 public developer-community threads; tenant hostnames and e-mails were rewritten before training, and a canary probe checks the finished model for memorised strings against a control run on the untrained base.
Quantised with mlx-node and mlx-lm, trained with mlx-lm (26 h over the rounds, 31.7 GB active memory, flat across every segment) and evaluated through Rapid-MLX, all on a single Mac Studio, with a chunked gated-delta training patch and full-state checkpoints so a killed segment costs minutes.
Twelve Forge apps LeanZero wrote for Jira and Confluence plus five forge-sim fixtures; manifests generated from the @forge/manifest schema and kept only when Atlassian's validator accepted them; UI Kit examples that compile against @forge/react 12; the Forge documentation itself, section by section, plus question-answer pairs kept only when the answer is a verbatim quote; endpoint and scope facts from six Atlassian OpenAPI specifications; 1,400 vetted question-answer pairs from LeanZero's community-research compendium and the answers we posted on the Atlassian communities; 220 public Atlassian Developer Community threads, question plus accepted answer; and the model's own generated apps, kept only when they passed the validator, the allow-list and tsc, plus a smaller set of manifests that passed the validator and allow-list.
Full fine-tuning a 27B needs several times its bf16 weights in memory, which no 96 GB machine has. A low-rank adapter trains on top of the frozen 8-bit base at 31.7 GB. At the end it is merged into the weights at 8 bits, where we measured that the delta survives: the merged model agrees with base-plus-adapter at 99.4% top-1. The 6- and 4-bit members are quantised from the merged bf16 weights; the smaller they get, the more of the delta is lost, and their cards print how much.