Making a Local-Model Swarm Predictable: 602 Commits, 100 Levers, and Three Bugs I Found Writing This (2026)
Mihai Perdum
Author
14 min readJuly 20, 2026
Key takeaways
The swarm engine went from just under 7,000 lines to 21,191, and from 18 tunable levers to 100. Zero levers were removed.
Every GOOSE_SWARM_* environment lever was silently OFF for the whole campaign: launching the app with open -n hands the spawn to LaunchServices, which discards the caller's environment.
Plan confidence is now min(draft agreement, spec clarity). One measured run scored 55 purely because three drafts agreed on 93% of files but chose 5, 6 and 7 subtasks.
Four false greens traced to one line: verified meant an oracle executed, not that the app was correct. The fix was a narrowed claim, not a new check.
Four levers shipped on mechanism evidence, not an A/B, because Fisher's exact on a 1-vs-1 table returns p=1.000 for every possible outcome and detecting a real effect needs ~46 runs.
Writing this post surfaced three live defects, including a learned skill that is loaded, announced, and then silently overwritten before the planner ever reads it.
Three weeks ago I wrote that a fleet of three small local models could be coordinated into producing correct, runnable software, and that the last measured limit had stopped being the swarm's coordination. That is still true. It is also not the thing I have been working on since.
The honest position today: this is nowhere near where I would use it day to day. What has changed is that it is starting to become predictable. That distinction is the whole point. Agents driven by local models are slow and unpredictable. I cannot do anything about slow, that is physics and a 27B model on a Mac. What I can attack is unpredictable, and it turns out most of the unpredictability was never in the models. It was in an engine that could not accurately report what it had done, running behind a UI that could not accurately report what the engine said.
Since 1 July: 602 commits, 682 files, +101,615 / -35,938. The engine file went from just under 7,000 lines to 21,191. Tunable levers went from 18 to 100. Not one was removed.
Note
Setup — three Macs on a LAN (M4 Max, M3 Ultra, and a third node), each running a qwopus3.6-27b-coder MLX build in LM Studio at roughly 20 GB, joined over LM Link. Goose Local Edition is a fork of Block's goose. The desktop app is Electron; the engine is Rust. Nothing here touches a cloud model.
One real build: plan confidence 100/100 after a clarify round lifted it from 84, three nodes generating, and a phase checklist where every tick is an engine event rather than a model's opinion
What was the engine actually lying about?
The first version of plan confidence asked the planner how confident it was. That produced a parseable score in 6 of 102 runs, and the calls that did answer usually ran long or hit their 900-second cap. A weak model cannot calibrate a 0-100 number about itself.
So it was replaced by something measurable: draft the plan several times in parallel, one per node, and measure how much the drafts agree. That has its own failure. A genuinely ambiguous request can score high just because the weak model happened to pick the same interpretation three times. The same vague spec was observed scoring 51, then 95.
The current formula is the minimum of two independently measured things: how much the parallel drafts agree with each other, and whether your spec actually pins down a product. Both appear in the panel as separate bars, with the engine's own reason string underneath each, because the number alone is useless for deciding what to do about it.
The clearest illustration is a build I watched start at 55 against a floor of 85. Here is the entire score, as arithmetic:
Three drafts agreeing on 93% of files score 55 because they chose 5, 6 and 7 subtasks. That is not a model being stupid. That is my scoring function having a cliff in it.
Then the causal chain that made it unfixable in that run. Agreement 55, clarity 30, so the score is 30, so it drops below the floor and asks me five questions. I answer all five. The rescore lifts clarity from 30 to 100. The score becomes min(55, 100) = 55. Agreement now binds, and agreement is the one thing asking cannot fix. Re-planning was off, so the loop never re-entered, the retarget that exists precisely to fix agreement never fired, and it built at 55 on a plan drafted before my answers existed.
Caution
The wrong turn — the engine's own stderr explained all of this at the time, and then told the user to set GOOSE_SWARM_ASK_REPLAN=1. Which brings me to the worst bug of the whole three weeks.
What did "verified" actually mean?
One line, and it accounts for four false greens:
rust
1final_verified = verdict.ran;
"Verified" never meant the app was correct. It meant an oracle executed on this tree, and ran is set true for any tree containing a single .py file. Meanwhile the smoke helper returns None on a spawn error or a timeout, and every call site correctly declines to raise a finding from it, because an inconclusive check is not evidence of a defect. So findings stayed empty, empty findings meant passed, and passed meant verified, on a run that executed nothing.
There are two green paths for a dead server. Either --help exits 0 without ever binding a port, or --help hangs, which is exactly what an entry point that ignores it and binds a port does, and the 30-second cap turns that hang into None, which raises no finding, which reports verified.
The fix is a narrowed claim rather than a new check. Two predicates where there was one:
passed means nothing I looked at was red. established means I genuinely looked. They are different questions and the engine only ever asked the first.
The reason the fix is shaped that way is a rule I now apply everywhere: never flip passed red, only ever narrow verified. Every countermeasure I have had to downgrade in this codebase was downgraded because its false positive drove a fix, and the fix loop then damaged a working app. An honest "unverified" costs a correct app nothing. A false red costs the whole run and hands a weak model a mandate to repair code that was already right.
Worth saying: the two cleverer designs I generated for this both came back flawed under review, and I shipped neither. One of them, a deterministic spec extractor, measured 0 out of 19 precision on real specs.
Why was every lever secretly off?
The desktop app is launched with open -n Goose.app. open hands the spawn to LaunchServices, and LaunchServices gives the app its own environment. So env FOO=1 open -n App sets FOO for open, which then exits.
I proved it directly rather than reasoning about it:
bash
1envGOOSE_SWARM_ENVPROBE=propagated open-n Calculator.app
2ps eww <pid># zero trace of the variable
Every GOOSE_SWARM_* variable intended for the desktop had always been discarded. The entire lever campaign ran with every env-gated lever off, and nobody noticed, because the one env value anyone bothered to check happened to equal the config value already sitting on disk. Meanwhile the engine was printing flag names at a user who had no possible way to comply.
The fix is not one line, it is a rule: config.yaml is the only channel that reaches the engine, so a lever with no config field cannot be turned on by a human at all. Every lever gets a desktop toggle, even the ones defaulted off. Precedence is env, then config.yaml, then the default, split into a pure function so it can be tested without env races.
And because nothing outside the process can reproduce that precedence chain, the engine now states its own resolved configuration into every run log as a levers_resolved event, computed by calling the same expressions it branches on. The comment above it is blunt about why:
Everything that tried, lied. The harness printed arm labels for a week while open -n discarded every one of them. Reading config.yaml back is not enough either, because the desktop provider force-sets six of these at spawn and env beats config. A number the engine did not emit is not evidence.
How do you tune something you cannot A/B?
This is the part I am most pleased with, and it is entirely negative.
I wanted to ship four levers about consulting the user instead of guessing. I could not A/B them. At roughly 25-37% base failure rate on this benchmark, Fisher's exact test on a 1-vs-1 table returns p=1.000 for every possible outcome, and there is a 37.5% chance an inert lever fabricates a win. Detecting a real effect needs about 46 runs. Each run is 45 to 110 minutes. That experiment does not exist.
So the admission bar changed shape. Instead of "does it measurably help", it became: a deterministic engine event proves it fired, and it is structurally incapable of making the app worse. Both, or it does not ship on by default.
The measurement that justified the whole set, same spec, three configurations:
Configuration
Open decisions
Asked
Invented
Reported confidence
Levers off (what shipped)
5
0
5
laundered 30 to 96
Levers on, ask cap 3
5
3
2 in silence
30 (honest)
Levers on, ask cap 6
5
5
0
30 (honest)
4 rows × 5 columnsHeader row enabled
That spec had exactly five items marked "DELIBERATELY NOT DECIDED, do NOT guess them". The shipping default guessed all five and reported 96/100 confidence about it. A related run left six product decisions open with the same instruction, landed confidence 90, asked nothing, and shipped a Swift app that guessed the opposite of my every stated preference: folders and tags instead of tags only, JSON instead of plain text, case-insensitive instead of case-sensitive search.
Two supporting findings from the same week, both of which are just embarrassing:
The research pass counted any successful builtin shell call as grounding, so an invented product answer preceded by one trivial echo scored grounded=true. And the retarget built each research question from the first 200 characters of the prompt, which on a wrapped turn was 171 characters of goose's own XML and 28 characters of real spec. Every research question that run asked was about goose's own turn-context wrapper.
What does the desktop actually show now?
I started this project on the CLI alone. Around 8 July I looked properly at the desktop app and fell in love with it, and 189 of the 602 commits have landed there since, across 20 shipped versions from 1.40.0 to 1.41.56 (my working build is further ahead at 1.41.88, unreleased). It is still far from where I want it, but the direction is right.
There is no socket. The engine writes an append-only JSONL event log and one live-rewritten digest per in-flight model call into .swarm/; the desktop polls that directory every 500 ms and folds it through a pure reducer. Every honesty property in the panel falls out of that shape, because the reducer can only assert what a deterministic event says.
1
The fleet strip
one row per physical node, not per task. It reads each node's live generation, and when the coder models draft in the <think> channel it counts those characters too, because a node with 10,794 thinking characters and an empty text channel was previously rendered as idle.
2
The status dot
driven by lms ps --json, LM Studio's own truth, not goose's digest. Green generating, amber processing the prompt, dim idle. If the probe fails there is no dot at all, rather than a confident wrong one.
3
The phase checklist
nine states, not three. A finished build task is unverified in slate blue, never green, and is only promoted to done when the end-to-end verify actually passes.
4
The metrics strip
elapsed is a fact; the ETA is deliberately a 0.5x to 2x band labelled "rough", because the single-node verify sink dominates the tail and a precise figure would be a lie.
5
The note box
type something while it builds and it is folded into the next dispatched worker, never into one already running.
The fleet strip alone took ten commits in about 36 hours. My favourite is the smallest: the live thinking rendered one token at a time (💭 always, 💭 ents) because the engine assigned the current stream chunk instead of appending to a rolling buffer. One push_str and it became readable prose.
The counter bug is the one worth learning from. For fifteen minutes of a real run, six of seven build tasks completed and the panel read "Build 0/7". The one time the number moved, it moved because work was added. The counter was asking "how much is proven?" while displaying the answer to "how far along is this?", and because the only Build row born done is the re-plan bookkeeping row, the numerator was literally the count of re-plans.
Why does the settings panel read like a lab notebook?
Because it is one. Every hint in it names a measured failure with a count.
Three nodes live off LM Studio's own catalogue, per-node task-share weights, and toggles whose help text is a paragraph each because a one-line rule cannot explain a quality-versus-speed trade
Those hints used to be truncated at the viewport edge, which produced strings like "This stops after a round that fails to beat the be…". The hint is the entire point of the panel, so it now wraps. And each group states what it costs you, in the same voice:
Before it says done — Buys: goose stops calling a broken app verified, 7 runs have. Costs: real minutes. Every check here RUNS something, and a failed check triggers a fix round.
A few of the levers, with the measurement that produced them:
The verifier's step budget. The final check builds the app, runs every command the spec advertises, checks the output, and fixes what is broken. It had the same 40-step budget as a worker that owns one file. Across nine runs, five sinks never reached their own verdict and three died on the cap, their last words literally "I've reached the maximum number of actions I can do without user input". The run still reported a result. A verifier that was cut off has not verified anything.
The spec-clarity probe. It runs alongside the planning drafts on the same nodes, and each node serves one request at a time, so on a busy fleet it queues and gives up. It died on 2 of 14 runs, and both times the engine fell back to cross-draft agreement alone and reported confidence 93-95 with zero questions, on the same spec where successful probes reported 30 and asked five.
The re-draft ladder. One run went 84, then 70, then 70, then 52 across three re-draft rounds, spent roughly 60 minutes of the entire fleet doing it, and shipped the round-2 plan anyway. Another spent 55 minutes growing best-of-N from 3 to 6 for drafts that were structurally impossible.
The second planning round. The backbone lock re-drafts the whole fleet a second time, about 250 seconds, to pin the consensus modules. Measured across 28 of 29 real runs, that round was never once adopted when first-round agreement was already 90 or above.
One thing the weights row deserves a correction on, because I got it wrong and reverted it 54 minutes later. Making a node's weight raise its concurrency oversubscribes LM Studio, which serves one request per model at a time. Observed live: workhorse got 3 tasks with 2 queued, mihai got 2 with 1 queued, and gabee sat READY with nothing. Weight now shapes routing share over time via work-stealing, and concurrency comes from the node's real capacity.
Does bringing Claude Code across actually work?
The import tool is the piece I am happiest with in practice.
Fifteen skills detected with their supporting-file counts; the amber badge counts files that are new or changed at the source since the last import, so a stale copy cannot masquerade as up to date
It also shipped one of the more humbling bugs. I re-imported my skills and nothing changed. The import had been a no-op on every existing skill since the day it first ran, and it reported success every time, because the copy refused any destination that already existed and the UI classified that refusal as "skipped" with a friendly toast reading "16 already present". Literally true, and the exact opposite of the useful thing to say: nothing was lost except every change since the first import.
Measured before touching code:
text
1atlassian-community-leanzero SKILL.md 23,210 b (Jul 13) -> 10,506 b in goose (Jul 4) 45%
2references/lexicon.md 62,053 b (Jul 16) -> 18,869 b (Jul 8) 30%
3leanzero-management SKILL.md 165,881 b (Jul 15) -> 41,052 b (Jul 8) 25%
4952 files at source, 766 imported: 186 missing + 26 stale.
Thirteen of the missing files were load-bearing scripts and references. The SKILL.md that did import instructs the agent to run scripts that were never copied. A skill that references files it does not have is worse than an absent one.
The fix half-worked, and the follow-up commit an hour later is the better lesson. I had verified the copy logic against synthetic files in /tmp and called it done. The real skills break it three ways /tmp never could: six of them are symlinks into other projects, so copying the link into a real directory throws; dereferencing everything instead inflates the tree from 952 to 3,082 files and dies halfway on a self-referential node_modules link, leaving a partial copy; and an inner node_modules symlink present on both sides makes cp refuse with "cannot copy to a subdirectory of self". Resolve the top level, skip every link inside it, and make the drift walk mirror the copy exactly, because a badge that promises what the button cannot deliver is just a different lie.
The same shared-skills-root move exposed something worse in goose's own discovery walk. On my real skills directory it found 3,082 files, of which 2,130 lived under node_modules, promoted two SKILL.md files vendored inside playwright-core into first-class skills injected into every system prompt, and made load_skill on one skill emit 928,497 characters, roughly twice the entire context window of the local fleet, from a call the system prompt invites the model to make. After capping the manifest and skipping dependency trees: 951 supporting files and 12,990 characters, 71x smaller.
What does it mean that goose writes its own skills?
A skill goose wrote about the fastapi stack after a build the engine proved compiled and passed its checks, with a disclosure that the lesson was phrased by a local model and can still be wrong
Two Swift builds each burned about 40 minutes of planning re-deriving identical knowledge, and the judge rediscovered "@MainActor on NoteStore" in both. The same lesson, paid for twice, thrown away twice.
So after a build that provably worked, goose reflects and writes a reusable per-stack skill. The design rules matter more than the feature. Only a deterministic engine gate may trigger a write, so the model never decides it did well, it only phrases what the engine already proved. It writes about the stack, never this app, because a cached decomposition carrying one app's features would drag the next one toward the wrong product. The stack key refuses rather than guesses, because React, Angular and a Node CLI all collapse to "TypeScript" and an Angular lesson poisoning a React build is exactly what this is meant to prevent.
And because a weak local model authored it, the only thing that makes it defensible is that you can see it and throw it out. That was not true at first: it wrote to a directory no skill-discovery root covered, so nothing goose learned about itself was ever visible. Moving it into the shared skills root is what made a collision reachable, so the write guard shipped with it. Authorship decides, not the path: only a file that is absent, or that carries goose's own provenance line, may be truncated. Strip that line while editing and goose will never rewrite it again, which is the harmless direction to fail in.
Sixty-four memories, each an entry in a plain text file, with a solid type chip and a grey imported:claude-code provenance chip that is simply the unrecognised-tag fallback
Memories came across the same way. The trigger for goose writing its own was measuring that it never had: zero remember_memory calls across 126,000 messages. Every memory on disk came from the import. The reason was a prompt that cancelled itself, telling the model to save proactively and then to always confirm with the user first.
What is still broken?
I found three things while writing this post, which is itself the argument for writing them.
The learned skill is loaded, announced, and then thrown away. The persona is pushed into the advisory research channel at swarm.rs:17946, and at swarm.rs:18064 the research phase does research_findings = findings..., a plain assignment rather than an append, unguarded. Research defaults to on, so on every default and golden configuration the sequence is: the persona_loaded event fires, stderr prints "reusing what worked on 1 previous build(s)", and the block is silently overwritten before the planner ever sees it. The telemetry says loaded. The prompt never contains it. My instinct that goose does not really use its skills yet turns out to be literally true, and it is one character class of a fix.
The confidence floor never reaches the panel. The engine emits ask_floor on exactly two events, run_started and plan_loaded. The desktop's only read of it sits inside the low_confidence_ask case, and that event does not carry the field. So the floor is always null, and the verdict falls back to the hardcoded band that two separate commits were written to abolish. The unit tests pass because they call the pure function with an explicit floor.
The secret scrubber is not protecting anything. There is a 742-line, thoroughly tested classifier with a refuse-on-hit detector for private keys, cloud tokens, JWTs, connection strings and high-entropy blobs. Its only reference in the entire repository is pub mod memory_classify;. It was landed deliberately dormant, and that is defensible, but the consequence is that the thing stopping goose writing a token into a file that goes into every system prompt is currently one sentence of English in a tool description.
What I will not claim
There is no clean end-to-end build-time comparison. The later benchmark runs recorded verdicts and line counts but stopped recording wall-clock, so I can tell you a specific lever saves about 250 seconds or that skipping the re-plan saves about 15 minutes, and I cannot tell you that builds got faster overall. I am not going to invent that number.
Almost every lever A/B here is n=1, and the one with the cleanest result carries a self-declared confound: the winning arm started from a higher-confidence plan than the arm it beat. The second pair that would settle it was never run. The skeleton-first lever was adopted on a wash under a "not worse, so default on" rule. The pillars lever tested 3/3 versus 2/3 on interface integrity and worse on tests, which is noise, so it stayed off.
And speed is still the systemic gap. Python apps run 40 to 47 minutes against a 15-to-25-minute goal. Recursive algorithm cores still defeat a 27B: handed the exact compile error three times, it could not fix an unterminated string literal.
Key takeaways
Most of what looked like model unpredictability was instrumentation that could not report itself. The engine now states its own resolved config and build SHA into every run.
Verified negatives beat plausible positives. open -n discarding the environment, a probe that died on 2 of 14 runs, a sink cut off on 5 of 9, a counter whose numerator was the re-plan count.
When the sample size to prove a lever does not exist, change the admission bar rather than faking the statistics: a deterministic event proving it fired, plus structural inability to make things worse.
A UI over an agent is a claims surface. A colour is a verdict, a request is not a fact, and a panel contradicting the engine is a false green in prose.
Publishing forces verification. Three shipped defects surfaced from re-reading my own code to describe it.
Next steps — the original write-up of the self-verifying swarm covers the architecture this post assumes, and Inside goose-swarm takes the scheduler loop and the judge thresholds apart properly. If you want to reproduce the grading discipline rather than the swarm, swarm-gym is the testbed that found most of these failures, and the MLX versus GGUF benchmark is where the runtime choice got settled. If you are trying to make agents behave predictably against a real system rather than a toy, that is most of what our engineering work actually consists of.
I still love this thing, which is probably obvious. What I want to know from anyone else running agents on local models: when your run reports success, what do you check before you believe it?
llama.cpp vs MLX on Qwen3.6-27B: MTP is 1.04x here, not 1.85x
Multi-token prediction is merged in llama.cpp and still an open PR in mlx-lm. I measured both on an M3 Ultra with the same model. Every default MTP setting was slower than no MTP at all, and the runtime that deletes the MTP head outright is still the fastest thing on the box.
