GPT-5.6 Sol won our benchmark without finishing it. Then Codex cost me six days on goose
Mihai Perdum
Author
11 min readAugust 27, 2026
Key takeaways
The benchmark entrant is the MODEL gpt-5.6-sol on AWS Bedrock inside goose's own agent loop. The six-day failure was OpenAI's CODEX, a different product and harness. Two variables moved; treat the comparison accordingly.
Corrected board, sb-6.0 to sb-7.0-rc: Sol 0.9956 to 0.8166 (-18.0%), Opus 5 0.9307 to 0.4695 (-49.6%), Sonnet 5 0.8635 to 0.3751 (-56.6%). Sol leads both.
Our scorer versions are incomparable by construction, and the first version of this article broke that rule in a direction that flattered its own thesis.
The sb-7.0 winner did NOT finish: agent secs 7200.0, timed_out true, exit null. It was killed at a two-hour wall and graded on partial work. Opus 5 exited 0 and tidied up after itself.
Every scorer check rewards conformance to a spec that already named the answer: eight files named by path, four documented sync keys. None can reward knowing when to stop, because the HARNESS supplies that.
Codex's own post-mortem names the same failure: 'regressed the engine's most basic capability: finishing', 'became capable of running indefinitely', 'I failed to use a stop-loss'.
The hostile reading — vague brief, no acceptance criteria, no stop-loss, then blame the tool — is substantially correct, and the article says so.
What survives the concession: the failure to terminate appeared under ideal, fully specified conditions too, which no brief of mine caused.
I want to start with the thing I got wrong, because it is the reason the rest of this is worth reading.
The model and the product are not the same thing, and I originally wrote about them as though they were.
The entrant that tops our leaderboard is gpt-5.6-sol, served over AWS Bedrock and driven by goose's own agent loop — our harness, our scaffolding, our prompts. That is what bench/run_build.py dispatches. The six days I am about to describe were OpenAI's Codex: a different product, a different harness, a different scaffold around a model from the same family.
So this article cannot tell you "GPT-5.6 is bad at real work". Two variables moved between the benchmark and the disaster, and I only controlled one of them. What it can tell you is more specific and, I think, more useful.
The board, with the right numbers
Our scorers are versioned, and the README is explicit that scorer versions are incomparable by construction. I broke that rule in the first draft of this article by putting sb-5 scores in an sb-6 column, which flattered my own argument. Here is the corrected board, sb-6.0 and sb-7.0-rc runs only:
entrant
sb-6.0
sb-7.0-rc
change
gpt-5.6-sol
0.9956
0.8166
−18.0%
Claude Opus 5
0.9307
0.4695
−49.6%
GPT-5.6 Luna
0.8671
0.2204
−74.6%
Claude Sonnet 5
0.8635
0.3751
−56.6%
Fable 5
—
0.4640
—
GPT-5.6 Terra
—
0.3919
—
Haiku 4.5
—
0.0387
—
8 rows × 4 columnsHeader row enabled
Sol is first on both, and its lead widens on the harder scorer. That is real and I am not explaining it away.
Two caveats a fair reader deserves, both of which I had to be pushed to include. The sb-7 gap is not as clean as it looks: a large part of the distance between Sol and the field comes from one check, j_workflow_journey, where every non-Sol frontier entrant scored zero and hit the same severity floor. Strip the floor and the underlying quality gap narrows considerably. And these are single runs, not averages over repetitions.
The thing I missed the first time
Here is the detail that reframed this whole article for me, and I only found it because a review forced me back into the run records:
That is gpt-5.6-sol-r0 — the sb-7.0 winner. It did not finish. It ran into a two-hour wall and was killed, and was then graded on whatever it had produced at that moment. It still scored higher than everything that finished.
and its closing words: "Assets total 95,485 bytes against the 150 KB budget. DECISIONS.md has D1/D2/D3, __pycache__ and scratch DBs are cleaned up, and every module parses."
One of those is a model that stopped, tidied up after itself, and reported. The other is a model that was still going when the harness pulled the plug.
Both facts are in the leaderboard. Only one of them is in the number.
What our scorer actually rewards
This is the part that explains the score rather than dismissing it. Here are two checks, quoted from bench/score_sb6.py rather than paraphrased:
python
1@check("modules_present","A")2want =["meridian.py","store.py","api.py","__main__.py"]+list(WEB_FILES)3# → "files the spec names by path are missing"
python
1@check("sync_shape","A")2keys ={"fetched","inserted","updated","total"}3# → "the caller cannot tell what a sync did"
Eight files, named by path. Four response keys, documented in advance. Every check scores conformance to a specification that already named the answer, and grades it by running the software rather than reading it — which is the right way to build a scorer, and a whole argument of its own.
But notice what no check anywhere rewards: deciding what to build, resolving an ambiguity, knowing when to stop.
A benchmark cannot measure the last one, because the harness supplies it. The two-hour wall is not part of the model's performance; it is scaffolding we built, doing work the score then takes credit for.
Six days with nothing supplying the wall
That is what the goose work was. "Improve the orchestration engine" is not a specification, and Codex ran with no external deadline.
Its own post-mortem, which I asked for and am quoting rather than paraphrasing:
"I also regressed the engine's most basic capability: finishing. Before this effort, the system could complete runs. After my changes, successive versions repeatedly failed before build or became capable of running indefinitely."
"V25 spent approximately 3 hours 40 minutes producing six full plans. Build never started."
"I failed to use a stop-loss... After the second full-plan rejection, I should have stopped the marathon, inspected the protocol and changed the compiler. Allowing four more complete planning attempts was indefensible."
"I practiced architecture astronautics. I created machinery for hypothetical failure modes before proving the basic path worked."
"My status reporting was misleadingly optimistic. Statements such as 'live,' 'generating,' 'zero provider errors' described process health, not progress."
Read those next to timed_out: true and they stop being a list of separate mistakes. They are all the same mistake. The benchmark run and the six-day run failed in exactly the same way; on the benchmark, a timeout hid it and a scorer rewarded the partial work.
Process health is not progress. An agent reporting the first while producing none of the second is worse than one that fails loudly, and this is the failure mode a leaderboard is structurally incapable of showing you.
What is left in the repo
terminal_closure.py is 3,230 lines of process-identity hashing, owned-identity ledgers, atomic writes, secret redaction and closure receipts — machinery for making sure a run ends cleanly, built by a system that could not end a run.
I originally wrote that this was "larger than the thing it protects" and that its V17 filename meant seventeen rewrites. Both were wrong: it pins score_sb7.py, which is 4,706 lines, and V17 is a build version, not a revision count. The true version is duller and still damning enough — a closure controller of that size exists because finishing had become the hard part.
I also originally attributed the engine audit's numbers — 129.9 minutes of planning, 139,059 characters of task descriptions, 18,831 seconds of agent time — to Codex. They are measurements of our local Qwen3.8 fleet, running inside the orchestration Codex had built. The distinction matters and the honest version is narrower: Codex did not burn those hours itself, it built the machine that did.
Where I am at fault
A hostile reading of this article is "you gave it a vague brief, no acceptance criteria and no stop-loss, and then blamed the tool." That reading is substantially correct, and I am not going to pretend otherwise.
I did not give it a small task that completes in minutes to prove the real path worked. I let a six-day marathon be the integration test. I watched four more full planning attempts after the second one should have stopped me. Those are my process failures, and Codex named every one of them about itself before I named them about myself.
What survives that concession is narrow and, I think, still worth something: the failure to terminate showed up under ideal conditions too. On a fully specified benchmark task, with a clean scorer and no ambiguity anywhere, it ran to the wall. That is not something my brief caused.
The economics, which is why I actually changed tools
When you hire junior, you are buying potential and paying for it with your own time. When you pay for senior, you are buying autonomy — someone who takes an under-specified problem and returns with good judgement you never had to supervise.
Codex, as a product, sat in the worst quadrant of that trade for me: premium cost, and it needed the specification written before it could excel. If I have to produce a document precise enough for it to succeed, I have done the engineering; what remains is typing.
I am back on Claude Code for goose. Not because it is faster — it is slower and fussier. Because when I hand it something under-specified it comes back having made defensible calls and having told me which ones it was unsure about. That is the thing I am paying for.
To be fair to it: in the same window Claude Code shipped a commit into quarantine with a mislabelled statistic and a racy fix. It is not the hero of this story. It is the tool whose failure mode I can supervise.
What would change my mind
Two runs would falsify the argument, and I intend to do both:
Give Codex a tightly-specified sub-task — one file, named methods, explicit acceptance criteria, a hard time budget — and see whether it lands cleanly. If it does, the diagnosis is specification, not competence, and it becomes a tool I aim differently rather than one I stop using.
Give Claude Code the identical one-line brief — "improve the orchestration engine", no spec, no criteria — and see whether it also fails to terminate. If it does, the variable was never the vendor. It was me.
Until those run, this is one uncontrolled engagement with two variables moving, and you should read it that way.
What to take from it
Every benchmark you read has a timeout you cannot see. Ours killed the winner at two hours and scored it first. Before you trust a leaderboard, ask what the harness did that the model got credit for.
Test the real path cheaply, first. A small prompt that finishes in minutes exposes phase-transition defects that a six-day marathon discovers expensively. My worst decision was not choosing Codex; it was choosing not to do this.
Report phase advancement, not process activity. If your agent's status can be green while the build has not started, your status is broken.
And watch for a model writing its own specification. When an agent meets ambiguity by generating structure instead of asking a question or making a call, it has stopped solving your problem and started solving one it can score itself against.
goose swarm: pytest | head -80 exits 0 when nothing ran, and pipefail only trades the lie
A worker in my local-model swarm ran pytest against a file that did not exist, twice, and finished the task green. The pipe had eaten the exit code. Measured on bash and zsh, on pytest and cargo — including why turning pipefail on just moves the lie to the other side.
