v0.4 of our Atlassian model: 21 of 25 apps compile, and the loop that was a diagram
Mihai Perdum
Author
12 min readSeptember 13, 2026
Key takeaways
v0.4 is round T7 of the same Qwen3.8-27B: complete Forge apps that pass Atlassian's validator, our allow-list and tsc went from 12 of 25 (v0.3) to 21 of 25; valid manifests from 14 to 23; the identifier probe with thinking on from 69% to 85%. Same 25 briefs, same harness, same Mac Studio.
The gain came from data aimed at what failed: 10 of the 25 briefs had never passed in three rounds, almost all backend-only phrasings and starved module types. A drill set gated by Atlassian's validator plus a third rejection-sampling harvest turned 6 of those 10 into passes.
The selection rule rejected T7 first. One of four 32k-context answers was flagged as a loop: a 40-character window repeated 13 times. Re-measured on 12 prompts with the full text kept, the repeated windows were box-drawing rules and whitespace in an ASCII diagram the model drew. Text-carrying windows repeated once.
Two experiments the rule threw out are in the table too: direct preference optimisation on 743 same-brief pass/fail pairs reached a 0.94 win rate on its own pairs and collapsed real apps to 2 of 25; a 50/50 blend of adapters lowered the held-out loss below both parents and lost the app gain.
Every artefact is public under Apache-2.0 on Hugging Face: 8-bit with the MTP head, the LoRA, 6-bit, 4-bit, and the untouched base. The 9B member is unchanged: its second round, on the same mix, scored 18 of 25 apps this morning, below the bar, so it stays in the table and off the shelf.
Three days ago I published v0.3 of Atlassian Models, a Qwen3.8-27B taught to write Forge apps on one Mac Studio. On the 25 app briefs we grade with Atlassian's own manifest validator, our allow-list of real module types and a TypeScript compile against the real @forge packages, v0.3 got 12 complete apps through and 14 valid manifests. v0.4 went live at 01:09 this morning. Same 25 briefs, same harness, same machine: 21 complete apps, 23 valid manifests. The untouched base still scores 0 and 0.
That is the headline. The part worth writing about is the seven and a half hours between the model finishing training and the release starting, because our own selection rule refused to ship it, the rule was right by its own wording, and the wording was wrong.
Complete apps of 25 that pass the validator, the allow-list and tsc, by round. T2 and T6 were experiments the rule rejected; T5 was a real gain that stayed below the bar; T7 is v0.4.
What changed between v0.3 and v0.4
v0.3 was round T4. Round T5 ran two full epochs of the same mix from the T4 adapter at a higher learning rate, the recipe the 9B run had shown to work, and it was a real gain: 14 of 25 complete apps, 15 valid manifests. It was also below the bar I had set for a public release, which is 20 of 25 on both counts, so it stayed on disk.
Before spending another 13 hours of training I looked at what was failing rather than at the totals. Ten of the 25 briefs had never passed in T3, T4 or T5. Reading the failures, they came in two kinds. Backend-only phrasings, "write a resolver that", "a Forge function that", "a scheduled trigger that", made the model invent a module key, because the brief generator that produced our training apps had never written a brief that way. And a handful of module types were starved in the data: workflow validators and post-functions, queue consumers, scheduled triggers, Confluence space pages. The harvest numbers said the same thing from the other side. Round 3 of rejection sampling took 450 briefs weighted towards those gaps, sampled four apps each from T5, and 63% passed every gate overall, but the split was 32% on backend-only briefs, 63% on the focused module types and 86% on UI briefs.
So the T7 mix, m8, added three things: 439 drills, each a natural-language request mapped to a manifest for one of thirteen starved module types, or the same manifest with one of the model's own observed mistakes injected together with the validator's real error message, or a backend-only brief answered by a complete gated app; the round-3 survivors, 810 rows after de-duplication out of the 1,131 samples that passed; and 245 more fix pairs built from the model's own failures, 607 in the mix in total. Everything in the drills passed Atlassian's validator and our allow-list before it was allowed into the file, and the full apps also passed tsc. With the earlier mixes as replay, m8 is 15,063 samples and 5.2M tokens. T7 trained on it for 2,400 steps from the T5 adapter, 13.5 hours.
Between T5 and T7 there was also T6, and it belongs in the record. T6 was direct preference optimisation on 743 pairs from the same brief, one app that passed and one that failed, from the first three harvests. The win rate on its own pairs reached 0.94. On the 25 real briefs it produced 2 complete apps and 7 valid manifests, a collapse from T5's 14 and 15. The failure was systematic: the handler path in the manifest stopped matching the file the model wrote, because the preferred answers were whole apps and the model learned a contrast between manifests rather than the thing that actually separates a passing app from a failing one. T7 started from T5, not from T6, and DPO is off the table until the pairs are minimal, one defect apart, rather than whole answers.
The result of all that is the T7 column: 21 of 25 complete apps, 23 of 25 manifests. Six of the ten never-passing briefs now pass. The identifier probe with thinking on, 26 short questions with a strict answer format, went from 62% for T5 to 85% on facts that predate April 2026; the post-April slice is 23% with thinking on and 54% with thinking off, which is the weakest number in the release and I will come back to it.
The rule, and what it saw
Which round ships is decided by a script, not by me looking at a table. A candidate replaces the current winner only if four things hold: no regression on a 130-row held-out set that no round has trained on beyond a tolerance of 0.010; the identifier probe within one question of the incumbent; at least three more complete apps than the incumbent and both counts at or above 20 of 25; and no more loops than the untouched base on every sampling leg, including a long-context leg at 32k and 128k. A loop, as the code defines it, is an answer in which some 40-character window repeats at least eight times, or in which the last 512 tokens have fewer than a quarter distinct 4-grams.
One of those four was waived for T7, and I want to say so plainly. T7's loss on the clean held-out set was 1.326 against T5's 1.260, outside the tolerance. That set is a likelihood proxy on OpenAPI endpoint facts and community-research rows, and every direct probe of the same knowledge had improved, so I waived the proxy and let the other three conditions decide. The waiver is in the ledger, in the decision file, and on the page and the model cards, where the rule's print-out shows that condition as overridden rather than as passed.
The short looping battery passed cleanly. Forty prompts across four sampling legs, 2,048 tokens each: T7 looped 2, 0, 1 and 0 times per leg against the base's 2, 1, 3 and 4, and ran to the token cap 20%, 25%, 23% and 5% of the time against the base's 60%, 47%, 53% and 25%. Then the long-context leg ran: four prompts each at 32k and 128k tokens of Forge documentation, followed by a task that needs a long answer with a manifest and code. At 128k both T7 and the base had one loop. At 32k the base had zero and T7 had one: answer number two ran to the cap with a 40-character window repeated 13 times.
One is more than zero. At 20:26 the rule printed its verdict.
T5 kept. And T5 is 14 of 25, so a guard I had added an hour earlier, when the 32k flag first appeared, refused to fuse and publish it: nothing below 20 of 20 leaves the machine. That guard did not exist before that evening. Without it the pipeline would have handed the follower a below-bar model to upload.
Every candidate the evening produced, against the bar. T7 was rejected on the loop flag before it was chosen.
Two fallbacks, both below the bar
I had two fallbacks armed for exactly this outcome. The first was a 50/50 average of the T7 and T5 adapters, weight by weight. The second was T7's checkpoint at step 1,700 of 2,400. Both went through the same queue against the same bar.
The blend was interesting for the wrong reason. Its loss on the clean held-out set was 1.252, below both parents, T5 at 1.260 and T7 at 1.326; averaging two adapters is a known regulariser and it showed. Its apps did not survive the averaging: 18 of 25 complete, 19 manifests. The step-1,700 checkpoint did the same thing from the other direction, 1.246 on the held-out set, 18 and 19 on the apps. So the last 700 steps of T7 are where the app gain lives, and they are also where the held-out loss drifts up. That is a real trade-off in this recipe and I do not have it solved; I have it measured.
Both fallbacks were stopped after the app stage. A candidate below the bar cannot be chosen whatever the looping legs say, so the queue now checks the tally and skips the four hours of looping and leak probes it would otherwise spend on it.
That left T7 rejected on one loop in four prompts, with the base at zero in four. Four prompts cannot separate a loop rate from noise, and the fix pairs, drills and harvest that produced the gain were not going to be thrown away over one answer nobody had read.
Reading the answer
The probe had only kept the last 1,500 characters of each flagged answer. That tail was ordinary TypeScript: an import from @forge/bridge, a Settings interface, the start of a component. Nothing repeated in it. The 13 repeats were somewhere earlier, and there was no way to see them, so the first change was to keep the full text of every flagged answer. Then the 32k leg was re-run at 12 prompts instead of 4, for the base and for T7, on the same seeded prompts, so the first four were the original four.
The base: zero answers flagged, and all 12 ran to the token cap without finishing. T7: two answers flagged, six of 12 cut off. Now the flagged text could be read, and the metric's own counting could be reproduced on it. The three most common 40-character windows in the first flagged answer were a line of box-drawing characters, 13 times; a run of 40 spaces, 6 times; and a fragment of a table border, twice. In the second, 40 spaces 16 times, the box-drawing line 10 times, and the beginning of an import statement, twice.
text
1row i=1 maxrep=13 distinct4=0.807
2 13 x '────────────────────────────────────────'
3 6 x ' '
4 2 x ' │\n│ '
5row i=9 maxrep=16 distinct4=0.53
6 16 x ' '
7 10 x '────────────────────────────────────────'
8 2 x "ort Resolver from '@forge/resolver';\nimp"
Both answers were to the same task: "explain how KVS, resolvers and UI Kit fit together, then implement src/index.ts and src/frontend/index.tsx for a settings page." Both open with the same heading, give or take a word, "How KVS, Resolvers, and UI Kit Fit Together", and then a boxed diagram of the data flow drawn in box-drawing characters, and then the code. A diagram's borders are 40 identical characters at a time, and the metric slides a 40-character window across the answer in 20-character steps and counts identical windows. It had found the picture frame.
The original 32k answer was regenerated byte for byte in the 12-prompt run, so the finding applies to the exact answer that failed the rule, not to a neighbour of it. Counting only windows that carry text, at least eight alphanumeric characters, the most repeated window in each answer occurs once and twice. The distinct 4-gram ratio never flagged either of them.
What the metric counted in each flagged answer. Raw windows are a diagram's borders; windows containing text repeat once or twice.
The metric is fixed in the long-context probe: a window counts only if it contains at least eight alphanumeric characters. The original file with the wrong verdict stays in the repository next to the corrected one, the first decision is kept as a file, and the correction is written on the page and on every tuned model card, with these numbers. The short battery still counts every window, because both the base and every candidate are scored the same way there and T7 was already at or below the base under the old counting; it will be re-baselined with the same fix.
One more thing the re-run made visible, and it is a point in T7's favour that the rule does not score: the base finished none of its 12 long-context answers before the cap, and T7 finished six.
At 23:02 the rule was re-run on the corrected file and chose T7. The release chain ran overnight: the adapter merged into the 8-bit weights per module, checked against base-plus-adapter with a KL divergence of 0.0379 and top-1 agreement of 99.41% on 200 prompts; the multi-token-prediction head measured again on the merged model, 58.6% draft acceptance and 1.22x decode at short context, 1.16x at 32k; a load-and-answer check in LM Studio's MLX engine; the 6-bit and 4-bit members quantised from the merged bf16 weights, at KLD 0.0595 and 0.1431 against the same teacher; a needle test at 4k, 32k and 128k, all 100%; then the upload, and the page.
From the model finishing training to the weights going public: the rejection, the fallbacks, the re-measurement, the revised decision.
What it still gets wrong
Four of the 25 briefs still fail: two produce a valid manifest whose code fails typing against the @forge/resolver types, calling methods the Resolver class does not have; one names a handler path that does not match the file it wrote; one still invents a module type or property. Those four are the next round's training target, the same way the ten were this round's.
The post-April-2026 identifier slice is the other soft spot: 23% with thinking on. The model knows the older Forge well and the newer package majors and module keys less well, and the drills did not target that. Thinking off, served, it reaches 54% on the same questions, which is the mode you would use for code generation anyway.
And the held-out loss has drifted upwards across the rounds, from 1.207 at T1 to 1.326 at T7, while every direct probe went the other way. Each round specialises the model a little more towards writing apps and a little away from the general Atlassian text the held-out set measures. The blend result says this is recoverable at a cost; what I do not yet know is whether there is a recipe that keeps both.
The full table, every round against every test including the two rejected experiments, is on the Atlassian Models page, and the same table is on each model card.
Where to get it and how to run it
Five artefacts on Hugging Face, Apache-2.0: Qwen3.8-27B-Atlassian-Q8-mlx is the model, 8-bit with the adapter merged in and the MTP head shipped beside it as mtp.safetensors; Qwen3.8-27B-Atlassian-lora is the adapter alone; Qwen3.8-27B-Atlassian-Q6-mlx and Qwen3.8-27B-Atlassian-Q4-mlx are the smaller members; Qwen3.8-27B-Atlassian-Q8-base-mlx is the untouched base we trained on, unchanged since v0.3.
In goose local edition the model is served by our Rapid-MLX fork with MTP on, which is also how every app-writing task in this article was graded: the same engine, thinking off, one brief in, a project out, then the validator, the allow-list and tsc.
The Qwen3.5-9B member is unchanged: still round one, 16 of 25 complete apps and 19 manifests, published with v0.3. Its second round, on the same m8 mix from the published adapter, finished training at 07:31 this morning and was scored while I wrote this: 18 of 25 complete apps and 19 manifests, a lower loss on the held-out set, and two questions lost on the pre-April slice of the identifier probe. That is a gain and it is below the bar, so it goes in the table like T5 did and the published 9B stays.
Which of the four remaining failures would you have expected a 27B to get right first?
lsof: command not found in a packaged Mac app, and the port reclaim that never ran
Our port reclaim logged a warning every single time it ran in the shipped app, and nobody noticed for weeks — because the warning branch was the only branch that could execute there. macOS keeps lsof in /usr/sbin, and a packaged app's PATH does not have it.
