I taught a 27B model to write Forge apps. As far as I can find, nobody had done that before
Mihai Perdum
Author
11 min readSeptember 10, 2026
Key takeaways
Atlassian Models is Qwen3.8-27B taught Forge, Jira, Confluence and JSM from real apps, the docs, the OpenAPI specs and our own community answers. The 27B's weights are on Hugging Face under Apache-2.0 in five artefacts, and a 9B member joined it this morning.
Measured, base to tuned: Atlassian identifier probe 15% to 69%; Forge manifests passing Atlassian's validator 0 of 25 to 14 of 25; complete apps that also compile 0 of 25 to 12 of 25.
The claim I will defend: as far as a search of Hugging Face and GitHub can tell, this is the first open-weights model trained to write Forge apps, and the first Atlassian-tuned model released with base-versus-tuned evidence on Atlassian's own validator. It is not the first model with Atlassian in its name; those exist since 2024 and I list them.
It was trained as a rank-32 LoRA on a frozen 8-bit base, 26 hours across the three kept rounds, on a single Mac Studio M3 Ultra with 96 GB. Every round, including the one that was rejected, is in the ledger and on the page.
It is a strong assistant, not a compiler. Use thinking on for questions and off for code generation; the 4-bit member was not re-scored on the app-writing tasks.
Qwen3.8-27B is a strong open model. Ask it to write an Atlassian Forge app and it invents a manifest format, imports packages that do not exist, and hands you something that looks right and cannot run. On our identifier probe the untouched base got the Atlassian-specific names right 15% of the time. On 25 Forge app briefs it produced zero manifests that Atlassian's validator would accept.
So I taught it. The result is on the Atlassian Models page and on Hugging Face under Apache-2.0: the same model, in the same file format, that now writes Forge apps which pass Atlassian's own validator and compile, answers questions about Jira, Confluence and Jira Service Management with the real identifiers, and does it locally on a Mac.
The numbers, base to tuned, all measured on one machine: identifier probe 15% to 69%. Forge manifests passing Atlassian's validator, 0 of 25 to 14 of 25. Complete apps that also compile, 0 of 25 to 12 of 25.
Is it a first? Yes, with a qualifier I can defend
I wanted to write "the first ever Atlassian-trained model" in the title. I searched before I did, and it is not true.
Hugging Face has carried a model called WizardLM-2-7B-Atlassian since May 2024; only a GGUF re-quantisation survives, its source repository is gone and its card says nothing about what it was trained on. There is a public dataset of Atlassian Community questions and answers from October 2024. Since late 2025 there are several small Jira fine-tunes: a 0.6B LoRA for Jira tool-calling trained on synthetic conversations, an adapter on Qwen2.5-7B whose name says JQL and whose card says nothing, a couple of Qwen3-4B fine-tunes whose cards say nothing about what they were trained on, and a set of Jira agent adapters trained on simulated trajectories. Atlassian itself says it hosts fine-tuned open models inside its own VPC; I found no released weights. None of these is Forge. Searches for a Forge-trained model on Hugging Face and GitHub return nothing.
So the claim I will put my name to is this: as far as a search of Hugging Face and GitHub can tell, this is the first open-weights model trained to write Forge apps, and the first Atlassian-tuned model released with base-versus-tuned evidence on Atlassian's own validator. It is also, as far as I can find, the largest, and the first built on Apple Silicon with the speculative-decoding head kept. If someone shows me a prior Forge model, I will update this paragraph.
What it read
The page lists every source, and the mix matters more than the recipe. Twelve Forge apps LeanZero wrote for Jira and Confluence, plus five forge-sim fixtures. Manifests generated from the @forge/manifest schema and kept only when Atlassian's validator accepted them. UI Kit examples that compile against @forge/react 12. The Forge documentation itself, section by section, plus question-answer pairs kept only when the answer is a verbatim quote from the docs. Endpoint and scope facts from six Atlassian OpenAPI specifications. 1,400 vetted question-answer pairs from our community-research compendium and the answers we posted on the Atlassian communities. 220 public Atlassian Developer Community threads, question plus accepted answer. And the model's own generated apps, kept only when they passed the validator, the allow-list and the TypeScript compiler, plus a smaller set of manifests that passed the validator and allow-list.
That last source is the one I would point at if you asked what made the difference. In round one the model sampled four apps for each of 300 fresh briefs, and 63% passed every gate. In round two it sampled four for each of 450 harder briefs, and 72% passed. Only the passers were trained on, together with "fix this manifest" pairs built from the failures whose sibling passed, with the validator's real error messages as the prompt. The final mix was 18,189 training samples with 1,581 held out, about 6.1 million tokens.
Nothing private went into the weights. Tenant hostnames and e-mails were rewritten before training, and a canary probe checks the finished model for memorised strings against a control run on the untrained base. The round-four adapter's probe came back with zero hits.
How it was trained, in one paragraph
Full fine-tuning a 27B needs several times its bf16 weights in memory, which no 96 GB machine has. So the base was quantised first, by us, from bf16 to 8-bit, with the speculative-decoding head kept as a sidecar file. A rank-32 LoRA then trained on top of that frozen 8-bit base at 31.7 GB of active memory, flat across every segment, using mlx-lm with a chunked gated-delta training patch and full-state checkpoints, so a killed segment costs minutes rather than the run. Three kept rounds: 11.3 hours for the first, 7.1 and 7.6 for the two that refined it, 26 hours of adapter training in total, all on the one Mac Studio. A fourth experiment, round two, was rejected and is on the page anyway. At the end the adapter was merged into the weights at 8 bits, where we measured that the delta survives: the merged model agrees with base-plus-adapter at 99.4% top-1. The tutorial on exactly how, with the configs, follows this post.
The rounds were not chosen by me. A rule in code decides whether a round replaces the previous winner, and it must hold on all four counts: no regression on the round-one validation set beyond 0.010, identifiers with thinking on within one question, at least as many complete apps passing every gate with manifests within two, and looping at or below the untouched base on every leg with non-termination not more than five points above it. Round two failed the first test at plus 0.018, and its one extra manifest, on inspection, came from invented identifiers. Round four passed all four and is what shipped, as release v0.3.
The ledger also records the day the rule was run wrong: the queue script passed no arguments to the picker, it compared round three against a column that did not exist and chose round one, and the chain was killed before any gate completed, the script fixed, and the decision re-run by hand. I mention it because "every number was measured" is only worth saying if the misfires are in the record too.
What you can download
Five artefacts, all Apache-2.0. The 8-bit merged model with the MTP head kept, about 31 GB, is the one to use. The adapter alone, rank 32, about 190 MB, for anyone who wants to train further from it. The untouched 8-bit base we trained on, rebuilt from bf16 with the MTP sidecar, so the adapter has something to load against. And plain 6-bit and 4-bit members quantised from the merged bf16 weights, about 21 GB and 15 GB, with the honest note that the smaller they get the more of the delta is lost: 98.26% top-1 agreement at 6-bit, 93.88% at 4-bit, and the 4-bit was not re-run on the app-writing tasks.
It runs in goose local edition, where the engine supervisor finds the sidecar and turns speculative decoding on, measured at 1.26 times faster decode on short context on the merged model. It loads in LM Studio with the MLX engine, and with mlx-lm directly. Those two read only the main weights and ignore the sidecar, so they run it without the speedup; the page explains why the file is laid out that way, and the base model's card says why you must never rename it.
A smaller member, Qwen3.5-9B on the same LoRA setup and the same data, in one run of two epochs, went public this morning under the same naming pattern, with its own adapter, base, 6-bit and 4-bit. Against its own untouched base it went from 15% to 62% on the identifier probe with thinking on, and it produced 19 of 25 valid manifests, more than the 27B's 14. That result sent the 27B back for a fifth round, two epochs at the higher learning rate the 9B used, which is training as I write this. A 4B had been on the list; it was dropped.
What it is not
It is a strong assistant, not a compiler. Fourteen of 25 manifests passing means eleven did not; twelve of 25 complete apps compiling means thirteen did not. The page prints the weak numbers next to the strong ones, and the identifier probe on the second slice, 23%, is one of them.
Use thinking on for questions and thinking off for code generation. That is not a style preference; on round one, manifest validity was 56% with thinking off and 16% with it on, while the identifier probe goes the other way. Version facts are pinned to the June 2026 wave of the Forge packages, and the weights do not update themselves. English only.
And the four days it took, 134 commits from the first scaffold on the evening of 5 September to the public release on the evening of 9 September, are four days on top of everything LeanZero had already written, answered and measured about Forge. The dataset was the hard part, and most of it existed before the model did.
If you build a Forge app with it, the validator will tell you whether it was right. That is the whole point of choosing a domain where the answer can be checked.
