Are AI benchmarks reliable? We had 32 models build the same app twice

Are AI benchmarks reliable? I think a single run on a long agentic coding task is a lot less reliable than the leaderboards make it look. On our Forge 2.0 benchmark, 32 AI models each built the same app twice, independently, and for the mid-range models one build's score can sit about 0.15 above or below that model's average (scores run from 0 to 1). Gemini 4 Argon scored 0.79 and then 0.29. Claude Sonnet 5.5 scored 0.59 and 0.55. Most of the argument about benchmarks is about contamination and gamed leaderboards. This is about the other part: how much the same model moves between two honest tries.

When Argon's second build came in, I asked the obvious question: why are some models capable in one session and not in the next, while others give you the same result every time? I had our AI agents read the transcripts of 16 models' builds, run the statistics on every pair and research every model's architecture from primary sources. Then I had an independent review attack the result. There's no flaky-model villain at the end of it. With two builds per model, you can't tell an erratic model from one that drew badly. Most of the swing comes from the penalties in our score, and very little from how much of the app got built. And part of one swing was our own harness quietly deleting the middle of a model's conversation.

Why should you care? You probably picked your coding model from a leaderboard, and many leaderboards publish one run per model. Your bills follow that choice every month. If the model at the top just drew well, you'll find out on your own tasks.

Two builds can't tell an erratic model from an unlucky one

That's the first of five findings, and each one has its evidence further down.

1. Two builds can't tell an erratic model from an unlucky one. For 24 mid-range models, the gaps look like what one shared noise level would produce. No single pair stands out once you allow for how many pairs there are.
2. Most of the swing is in the penalties. Our score multiplies how complete the app is by two penalties, and those penalties carry most of every big gap.
3. How differently two builds behave barely predicts how differently they score. Two near-identical builds can be far apart, and two very different builds can tie.
4. Often the swing is one defect the model's own tests didn't expose. The model tested on its development site, the bug only showed up on our scoring sites, and the scorer then charged it more than once.
5. Part of one swing was our harness. It cut a model's conversation without telling anyone. We fixed it and re-ran that model.

Architecture (hybrid attention, mixture-of-experts sparsity) doesn't separate the steady models from the swinging ones, at least not with this data.

How Forge 2.0 builds and scores an app

Forge 2.0 is our benchmark for AI coding agents. Forge is Atlassian's platform for apps that run inside Jira and Confluence. Each model gets a working Jira app written against an old (v1) contract. It has to upgrade it to v2: migrate the stored data, check permissions per person, respect rate limits and add dashboards. The model works inside goose local edition, our fork of the open-source goose coding agent, as a single agent with a budget of 300 model calls. goose asks for "medium" reasoning where a model offers that level, and otherwise switches reasoning on at the provider's default. Most models are reached through OpenRouter, a service that routes one API call to whichever provider hosts the model.

When the model is done, the scorer deploys what it built into our Forge emulator and runs it against three seeded Jira sites. It runs 105 checks, each one automated test of one behaviour ("migrated rows keep their source"), scored from 0 to 1 and grouped into 20 tiers. The final score multiplies three things (the formula is in the scorer's header, score_forge2.py, read at commit c373846, and I've written up how that scorer works before):

• completeness: how much of the app works, from 0 to 1.
• the reliability factor: in each of 19 groups of checks, the worst check costs up to 10% of the score. However badly a build does, this factor never drops below 0.25 (I'll call models stuck there "floor-pinned").
• criticals: five classes of release-blocking defect (a corrupted migration, a webtrigger that accepts what it shouldn't, an admin hole, a permission leak, a duplicated write). Each class found multiplies the score by 0.6, once.

Every check keeps its worst result of the three sites, except a few excellence checks (economy ratios, round trips) that keep the mean. The scorer's comment says why: "a correct app is correct on every site, one that happens to fit one pack is not." That rule matters more than I expected. 52 of the 60 builds we have per-site scores for end up below every one of their own site scores, and 2 tie. Jev Router's weaker build scored 0.64, 0.87 and 0.97 on the three sites, and its final score is 0.6398, right at its worst.

The scoring sites are seeded and emulated, so most of the scorer repeats exactly for a given app and site. A few timed checks (boot counts, paint clocks, settle windows) run on the machine's wall clock, which is why we never score two builds at once, and we haven't measured how much those move on a re-score. The simple cases do repeat exactly: two builds that left the v1 starter untouched both scored 0.0617, and two whose apps never deployed both scored 0.0014.

What do the numbers mean as apps? Argon's 0.79 build migrated every old row on all three scoring sites (59, 66 and 100 of them) with its source intact. Its 0.29 build corrupted the migration. Each model gets two independent builds, the board publishes the median and the range, and the methodology page has the rest. The serving route wasn't identical for everyone. Argon ran direct on Google. goose ran Mistral Large 4 and MiniMax on my laptop instead of our Mac Studio (still through OpenRouter, each served by its own vendor). For most of the others, OpenRouter chose among several providers. Hy4 preview and the first DeepSeek V4.1 Flash even changed provider inside one build.

AI agent reliability: one build is worth about ±0.15

So are the swinging models more erratic, or does every model share the same noise and some just drew far apart?

You can't measure a model's variance from two samples. I tried anyway, by pooling. I took the 24 models served by OpenRouter or their vendor whose two-build mean sits between 0.15 and 0.65. Floor-pinned models can't swing much, so I left them out. I also left out our own local fine-tune, which ran on a different machine and engine (putting it back changes nothing). The pooled within-pair standard deviation is 0.153.

If those 24 models all had that same noise level, you can work out how often a gap of a given size should appear (statisticians call it a half-normal distribution). On the 20 October data, it predicts this:

Argon's 0.50 has about a 2% chance for any single pair. But there are 24 pairs, and across 24 a gap that large shows up somewhere about 4 times in 10. As far as two builds can say, Argon is the tail of a shared distribution. With Hy4's re-run swapped in for its old pair, the pooled SD is 0.146 and Argon's chance becomes about 3 in 10, so the conclusion holds.

What does a single number on a board mean, then? A gap of 0.2 to 0.4 between two builds is normal for a mid-range model on this task. The median of two is the number to publish, with the range next to it as the error bar. Separating models by how consistent they are would take tens of runs each. Nobody pays for that on a 300-call task, us included.

Other people have measured the same thing. A September paper, Identical Runs, Different Results, found that identical runs of one coding-agent setup "varied more than the pairings differed from one another". On Randomness in Agentic Evals reports that "single-run pass@1 estimates vary by 2.2 to 6.0 percentage points depending on which run is selected, with standard deviations exceeding 1.5 percentage points even at temperature 0." Princeton's Towards a Science of AI Agent Reliability found that "recent capability gains have only yielded small improvements in reliability." And Anthropic's guide to agent evals has the arithmetic: a 75% per-trial success rate passes three trials in a row about 42% of the time.

Where the gap comes from: penalties, not completeness

The score is completeness times the reliability factor times the critical penalty. Taking the log of the high/low ratio turns that product into a sum. So each part's share of the gap can be read off exactly. The 13 largest swings, on the 20 October data:

Summed over these 13, completeness carries 28% of the gap, the reliability factor 46% and criticals 26%, so 72% comes from the penalties. Leave out Hy4's retracted pair and it's 80%. For most of these models, the two builds' completeness differs by less than a quarter. Muse Wonder's weaker build was even slightly more complete (that's the −4%), and the penalties did all of the damage and then some.

Why does the score amplify like this? A build that's a little sloppier in four groups loses about a third of its score through the reliability factor (0.9 to the fourth power is 0.66). A critical is a flat x0.6, however small the code behind it: one extra response field (Grok), one missing who: 'user' argument (Qwen Omni), one write in the wrong code path (Argon). And one root cause often pays more than once. Argon's migration bug fired the migration critical, failed a second migration check and cost a reliability group too. The scorer counts criticals once per root cause, but the reliability factor charges a cause in each group it fails, by design. Its docstring: "one cause that fails rows in two groups pays in both."

I think that severity is right. A permission leak or a corrupted migration IS a release-blocking defect, and you should know which build has one. It does mean the score gap overstates how different the two apps are.

Row distance doesn't predict score distance

So I counted how differently the two builds behave. For each pair: the number of the 105 checks whose score changed by 0.5 or more, and the total difference summed across all 105. Call it row distance. One root cause can zero many checks at once (Mistral's unfinished backfill zeroes 18), so it isn't a pure count of different decisions. I expected a strong link. There isn't one.

Over the 30 pairs where both builds produced an app, row distance and the score gap correlate at r = +0.17 (95% interval −0.20 to +0.50) on the 20 October data. With Hy4's re-run it's +0.09. Both are weak. Neither is significant.

So "unpredictable" comes in two kinds. Some pairs differ on many checks: Hy4's old pair, FlashX, MiniMax, Haiku, Ling, Step 5 and Mistral. Near the bottom of the board the 0.25 floor hides that difference. Haiku 5.5 differs on 27 checks and still scores the same twice. Other pairs agree almost everywhere except one decision, and that decision trips a critical or a few reliability groups. Argon is the clearest: its two builds score identically on 88 of 105 checks.

The checks that flip most often across all 32 pairs are edge-case diligence: following a re-estimate, sorting the ledger, posting a comment as the user, healing a dropped record, setting a sprint's initial membership. They flipped in 15 to 17 of the 32 pairs. You only get those right if you go looking for the edge cases.

Often the swing is one defect the model's own tests didn't catch

For the forensics, our agents read the goose session databases, the per-call telemetry, the request logs and the scorer verdicts for 16 models, in three separate passes. Five moments from those builds say more than the tables.

Gemini 4 Argon wrote a "migration complete" record from a page that should only read

The weaker Argon build put a write inside the admin page's read path. Opening the page called getMigrationProgress, which called ensureMigrationCatalog, which saved a migration record with this line (src/sync.js:282):

1complete: v1Keys.length === 0,

Before the first migration run there are no v1 keys in that record, so the page saved a migration marked complete. The scorer opens the admin page first. Argon's tests never did. From then on the migration showed "Migrated 0 of 0 v1 rows" and the v1 rows were rebuilt with the wrong source. On the worst scoring site the scorer found "5 of 66 v1 rows in scope-ledger with the source attributes recorded by the probe; preserved attributes changed 61", and the next check found the "migrated v1 payload corrupted at 371 checkpoint row(s)". The rows survived, pointing at the wrong source.

Argon's final message said: "Verified 88/88 v1 rows migrated, 0 killed or failed invocations, 0 scope-status mismatches across all 978 live issues". On its dev site that was true. Its own screen read "Migrated 88 of 88 v1 rows (complete)". Its self-tests all ran the reset and the scheduled reconcile before opening the admin page, so the order that broke it never happened on its machine.

Argon's other build did better. I'm not sure it meant to. Its getMigrationProgress() returns { n: 0, total: 0, complete: false, text: 'Migrated 0 of 0 v1 rows' } without writing anything when there's no state yet, and the scorer found "all 59 v1 source rows intact". The trace analysis found no reasoning in either build about the page opening before the first run, so my guess is the stronger build's safety was incidental. Two builds can't tell you how often Argon makes that choice.

Qwen 3.8 Omni Flash dropped one argument, and the dev site couldn't notice

In its weaker build, Qwen 3.8 Omni Flash fetched issues for a person's view without saying whose permissions to use. Its own request helper (src/jira.js:206) decides like this:

1const who = opts.who === 'user' ? api.asUser() : api.asApp();

No who, so the check ran as the app. The app can see everything. The scorer found "hidden issue data shown to a person": a permission leak, x0.6.

The model thought about it, too. Late in the build it wrote: "The dev site apparently lets every user browse everything (hidden 0 for all). The contract says the scoring site has hidden issues. My visibleIssues path uses bulkfetch as the user, which is the documented mechanism; if the dev site grants all, hidden stays 0. That's fine — the mechanism is right." It believed the request ran as the user, and on a site where everyone can see everything, correct code and buggy code both print "hidden 0".

The stronger build passed the identity explicitly (jiraJson('user', route...)). It also built an independent checker that rebuilt the numbers straight from Jira's changelogs and compared them with what the app stored. That checker first came back "VERIFY: PROBLEMS — 14 sprints, 628 issues, 212 true changes", the model fixed what it found, and it ended on "VERIFY CLEAN". I'd want that habit from any agent I pay for: check your result against something you didn't write yourself.

Grok 4.7 returned one field too many, then repeated itself until the output ran out

Grok's weaker build had a person-scoped view function that ended like this (src/views.js:77):

1return { sprint, ...totalsView(totals), hiddenCount: live.length - rows.length, changes: rows, members };

The stronger build returned the same object without members. That one field leaked hidden issue keys, and the leak critical fired only in the weaker build.

The weaker build had a second, stranger problem. Four times, it repeated a sentence or two over and over until it hit its 65,536-token output limit (its stronger build did this once). The first starts: "The field write batches by value, not by issue. I'll split each batch so no request touches more than 200 issues.The field update batches by value, so one request touched 342 issues. I'll split each batch at 200 issues.The field write grouped by value..." and goose marked the reply "[OUTPUT TRUNCATED: the model hit its output-token limit mid-generation — this response is INCOMPLETE]". Those runaway replies stayed in the conversation. From call 182, OpenRouter's compression started cutting the middle of it, and the last ~120 calls made no edits.

Compare one decision in each build. The weaker one, trying to save quota: "The candidate search is the expensive part. I'll check whether a narrower query still finds every change." The stronger one, looking at a similar filter: "That filter would skip removals. I'll keep full changelogs on the first run and, after that, only for issues updated since the watermark." The weaker build scored 0 of 15 on finding removals.

Hy4 preview worked without most of its own history

Hy4 preview's weaker build was identical to the stronger one until about call 30. Then its prompt collapsed. On 268 of 301 calls, the model saw roughly 5,000 to 9,000 tokens while the real conversation grew from 95,000 to 905,000. You can watch it lose the thread:

Wait — I haven't created src/sched.js yet! I was planning to. My earlier messages described the plan but I haven't written any files.

Later it remembered a summary that never existed: "I have a summary of prior work but context was reset; the summary says: manifest.yml rewritten with all module keys, src/ rewritten with 22 modules, lint clean". It re-read the starter brief more than 50 times, repeated identical reasoning 53 times, and announced "Everything is green (lint clean, 122/122, 200 smoke)" over a tree that was still v1. Near call 296 of 300 it found out: "The manifest is still v1! So we haven't yet updated manifest.yml."

That build scored 0.0555. The untouched v1 starter app scores 0.0617. From the trace, I think most of that was us, and the harness section below shows how.

Claude Sonnet 5.5 said what it hadn't checked

Sonnet 5.5 is my reference point for a steady pair. It's boring, in a good way. Both builds followed the same plan, the same file set and the same order, and made some of the same mistakes. 80 of 105 checks are identical, with 2 flipping between pass and fail. It used 59 and 78 calls of its 300 and had no critical in either build. And its hand-off did something almost no other model did:

I only exercised part of it on the dev site. npm run lint reports no issues and the three Custom UI surfaces build. I did not run the full 6-hour sequence, and several guarantees are untested.

Its other build ended with "passes everything I could check on the dev site. I did not cover the world-change cases (§12) or the 2-hour migration deadline", where §12 is the task's section on sprints changing mid-run. The scorer agreed: the sprint-close check it admitted skipping is 0 in both builds. Most models that declared done overclaimed, by our reading of the transcripts. Sonnet listed its gaps, and it scored 0.59 and 0.55.

Most decisive bugs were ones the dev site didn't reveal

In about seven of the twelve swings we traced (Argon, Qwen Omni, Qwen3.8 Flash, DeepSeek V4.1 Flash 1017, MiniMax, Jev Router and Muse Wonder), the decisive defect was one the model's tests on its development site didn't show. Sometimes that was the size of the scoring sites. The dev site holds 88 v1 rows, at most 28 per sprint, while the largest scoring site holds 100, and Jev Router (a router that sent almost all of its calls to GPT-6.1 Sol) hit 47 storage size errors on the larger site after storing each sprint as one growing value. Sometimes it was order or permissions, as with Argon and Qwen Omni. Muse Wonder capped migration at 20 rows per sprint per run. The cap fired on the dev site too, and its own counter showed the race. But nothing on the dev site checks a row's source, so the 20 rows written with the wrong source only showed up on the scoring site. GLM 5.3 FlashX, Grok and Gemini 3.8 Flash had two or more independent causes.

The task's hourly quota (2,400 points, 1,680 for background work, set by the task's own rate model, not by Atlassian) was the trigger in at least four pairs: MiniMax, DeepSeek 1017, Mistral Large 4 and Gemini 3.8 Flash. Qwen3.8 Flash's weaker build had a units bug, sleepUpTo(wait, wait), that slept milliseconds where it meant seconds and retried 58 times inside a 30-second Retry-After window. The stronger one wrote sleep(retryAfter * 1000 + 60).

The deciding moment came early or late, with no rule I can see. DeepSeek 1017's weaker build switched to a costlier hourly-sync design at turn 69 of 263. Jev Router's weaker build made its storage choice around call 19 of 75. Muse Wonder's cap came at call 90 of 117. GLM 5.3 FlashX's second bug, a variable shadowing an imported function, came at call 219 of 230 (its criticals came from an earlier webtrigger design), and after it the queue consumer crashed 22,077 times while the model declared done.

Does architecture explain it? Mostly no

Attention design doesn't separate the groups. Linear-hybrid models show up among the steady ones (Ember-1, Qwen3.8 Max Prime), the low-scoring ones (Ling 3.1 Flash, GLM 5.3 Flash) and the swinging ones (Qwen3.8 Flash, Qwen 3.8 Omni Flash, GLM 5.3 FlashX). Sparse attention covers steady GLM 5.3 Prime, Hy4 (0.06 apart once our bug was gone) and swinging DeepSeek 1017. Max Prime's and GLM 5.3 Prime's designs are inferred from their base models, since neither has its own model card. Dense against mixture-of-experts doesn't help either, because nearly every model here is MoE.

The one contrast worth watching is inside Z.ai's family. Z.ai's docs describe GLM 5.3 Flash as "320B total parameters with 18B activated" and "the first open-source frontier model to combine sparse and linear attention". FlashX, named alongside it, swings, while the full-sparse GLM 5.3 and 5.3 Prime don't. There's literature for a mechanism: Jelassi et al. found that "transformer models dramatically outperform state space models at copying and retrieving information from context", and MiniMax explained why M2 went back to full attention partly because "linear attention is currently far more sensitive to numerical precision than full attention". But every hybrid model that swings here is also a small "Flash" tier, so I can't separate tier from architecture with this data. Hypothesis only.

Google, xAI, Anthropic, OpenAI and MiniMax don't publish parameter counts, and neither does Meta for its closed Muse models, so for 11 of the 32 models there's nothing to compare anyway.

I found the effort setting more telling. "Medium" reasoning effort often isn't a real level. DeepSeek's own docs map medium to high. Mistral documents only high and none. Hy4's chat template accepts only high or no_think and raises an error on anything else, so for Hy4 goose sent no effort at all and just switched reasoning on, as it did for Ember-1, MiMo, GLM 5.3 Prime, FlashX and Qwen 3.8 Omni Flash. OpenRouter maps an unsupported effort "to the nearest supported level", without saying which way medium goes when the choice is low or high. So "medium", where we send it, is a request. How deep the model actually reasons is up to the provider. Temperature isn't the lever: goose sends none, so provider defaults apply.

I'd look at the long horizon instead. I think a lot of the swing starts there. Over 300 calls, a model that's right 99.9% of the time per call finishes clean about 74% of the time (0.999^300), and one that's right 99.5% of the time about 22% (0.995^300). A 0.4-point per-call difference becomes a 3x gap at the end. And models get worse once their own errors are in context: "models become more likely to make mistakes when the context contains their errors from prior turns." That fits Grok's runaway replies sitting in its conversation, though I can't prove they caused the rest.

Infrastructure noise in agentic coding evals: our own harness

Anthropic wrote about infrastructure noise in agentic coding evals in February: in internal experiments, "the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01)". Ours was a single line in goose's OpenRouter provider, inherited from upstream goose in August 2025. Our DeepSeek V4.1 Flash 1017 write-up first named it as a suspect, and this study traced it.

How OpenRouter context compression (middle-out) cut Hy4's context

The line added transforms: ["middle-out"] to each OpenRouter request, with the comment "Always add transforms: ["middle-out"] for OpenRouter to handle prompts > context size". OpenRouter's docs now call the feature context compression and describe it as "removing or truncating messages from the middle of the prompt, until the prompt fits within the model's context window." It's on by default only for endpoints with 8,192 tokens of context or less. We shipped a goose that asked for it on every request, so this one is on us.

The problem sat in how goose decides when to compact. It read the provider's reported prompt size as the size of the context. With compression on, the reported size is what OpenRouter kept, so the cut hid itself. Grok 4.7's weaker build was billed 207,145 prompt tokens at call 181 and 76,843 at call 182, while goose kept sending about 2.1 million characters and its turn context still said "of 500,000 tokens". goose never compacted, because as far as it could see the conversation was small. Hy4's weaker build was served 5,000 to 9,000 tokens of a 95,000 to 905,000-token history on 268 of 301 calls.

Grok's stronger build reached 256,000 billed tokens untruncated, so its real cut point sits above that. Both builds of Glimpse, another model on the board, show a single drop to 15,000-20,000 tokens with no goose compaction marker, which might be the same thing, but nobody traced them. Mistral Large 4 and Ling 3.1 Flash also show prompt drops, and those are goose's own compactions ("Conversation compacted · 210k → 13k" in the transcripts).

What we changed, and what Hy4 scored after

goose 3.0.125, released on 20 October, turns it off. From the release notes: "Middle-out is off by default. A new setting, OPENROUTER_MIDDLE_OUT (a switch beside the OpenRouter host picker), turns it back on." A prompt that doesn't fit is now refused, and goose compacts and retries. The same release stops a reply when its last 32,768 characters are one block repeated exactly, which is Grok's runaway.

We removed Hy4's old pair from the board on 21 October and re-ran both builds on 3.0.125:

The gap went from 0.33 to 0.06. Most of the swing went with the bug. Both builds finished well inside the budget, the prompt grew call after call with no drop, and the two builds differ on only 3 checks (as close as Opus 5.5's pair). Hy4's median on the live board is now 0.4852, ninth, where the old median was 0.2228. Ninth is a loose rank: Hy4 sits in a different comparison group from the eight above it, and it ran on a newer goose than they did.

I can't fully separate the causes, because the re-run changed two things at once. Compression was off, and OpenRouter was also pinned to Tencent's own endpoint with no fallbacks (the old pair had been routed across DeepInfra and Makora). It also ran on the laptop instead of the Studio. I think the truncation explains most of the 0.26 the median moved, given how the old weaker build lost its history. I can't prove it. The re-run can't say which change did what.

The steady models share a serving path and room under the budget

They had one serving path for both builds: first-party for Opus 5.5, Sonnet 5.5, GPT-6.1 Sol and Qwen3.8 Max Prime, Fireworks for Ember-1, Alibaba for GLM 5.3 Prime, Makora for MiMo v2.6 Flash. That's necessary, not sufficient, since most of the swinging models also had one path.

And they had room under the budget. GPT-6.1 Sol used 47 and 48 calls of 300. Sonnet 59 and 78. Opus 115 and 133. MiMo 195 and 188. GLM 5.3 Prime is the exception at 295 and 298. Grok finished one build and ran out on the other, and both MiniMax builds used all 300. Near the limit, one early choice that costs 30 extra calls turns into a cliff.

MiMo and GLM 5.3 Prime look steady and aren't quite. MiMo's two builds have different module layouts and mirror-image per-site scores, and the worst-site rule pulls both to about 0.5. GLM 5.3 Prime's lower-scoring build is actually the better app (0.67 before its critical): ordering the backfill before the migration trips the same critical as Muse Wonder.

The floor-pinned models (Haiku 5.5, Beam, Muse Spark Open, Qwen3.8 27B, GLM 5.3 Flash) agree mostly because of the floor. They sit at the 0.25 clamp in both builds, either stopping early (16 to 126 calls) or, like GLM 5.3 Flash's weaker build, running to the budget.

Where we were wrong

We published Hy4 preview's old pair, and it measured our truncation as much as the model. It's fixed and re-run.

We send "medium" where a model offers it and the provider's default where it doesn't, and even "medium" doesn't mean the same thing to every model that accepts it. The board should say which level each model actually got.

The reliability factor charges one root cause in each group it fails. We designed it that way. After this study I think we should revisit it, because it makes one bug look like four.

And two builds per model is too few to rank models by consistency. A third build cuts the standard error of the mean by about a fifth. Cutting the noise by a third takes four or five builds. It costs money (most run pages show what that build cost; Argon's, run direct on Google, don't), and for now we've gone cheap. Our to-do from this: publish the row distance next to the range, flag any build whose context was cut, and revisit the reliability charge.

Are AI benchmarks reliable enough to choose a model?

For choosing between models that are 0.3 apart, I'd say yes. For ranking models inside 0.15 of each other from one or two runs, no.

If you're picking a coding model from any board, ours included, read the median and the range together. Run your own task at least three times on the two or three models you're choosing between, and compare medians. Check whether each run finishes well inside its call budget. A low row distance alone wouldn't have protected you from Argon, which differed on only 5 checks.

If you build agents: in several of these pairs, the difference between the good build and the bad one was whether the model tested the path that broke, as another user, on more data, or against a check it didn't write. A harness that pushes for that is the lever I'd try first. And check what your agent sends to OpenRouter. Search the request body for transforms or context-compression. If compression is on, compare the usage.prompt_tokens the provider reports with what you actually sent. To turn it off, stop sending transforms: ["middle-out"] (or an enabled context-compression plugin) and send plugins: [{"id": "context-compression", "enabled": false}], and if the conversation doesn't fit, fail loudly and compact.

goose local edition isn't a polished product yet (we use it because we know what it's made of), and this study found a line in it that had been asking OpenRouter to cut over-long conversations for over a year. If you've run the same model twice on a long task, how far apart were your two runs? And if your agent goes through OpenRouter, did you find middle-out in its requests? I'd like to compare notes in the comments below. Each build on the board has its own page on the Forge 2.0 board, next to our earlier write-ups of the October board, Jev Router, DeepSeek V4.1 Flash 1017 and Mistral Large 4.