Jev Router topped our coding benchmark. 117 of its 118 calls went to GPT-6.1 Sol

TypeSafe's Jev Router (typesafe/jev-router on OpenRouter) went to the top of our Forge 2.0 agentic coding benchmark this morning. Its two builds scored 0.6398 and 0.8523, a median of 0.746, ahead of Claude Opus 5.5 at 0.6745. OpenRouter's own generation records show where the work went. Of the 118 calls the router handled, 117 went to GPT-6.1 Sol and one to GPT-6 Astra. GPT-6.1 Sol's own pair, 0.6495 and 0.5371, sits third.

So the model that came first was, on this task, the model that came third, called through a router. I saw the board and thought it was a bit crazy. My first question was the obvious one: what is Jev anyway? Jev is TypeSafe's decision model, and it writes no text. Jev Router uses it to pick a model and a reasoning effort for each request, the picked model writes the answer, and you pay that model's price.

The second question was how a router that kept picking a third-place model can end up first. Sol's pair ran five days earlier on an older build of goose, our agent. And the router may have changed Sol's reasoning effort on some calls in a way our records don't show.

On our task, a router that sent 117 of 118 calls to one model behaved like that model, plus whatever effort it chose. Two long agentic builds per model are too few to rank things that close. If you want GPT-6.1 Sol for this kind of work, I'd call Sol directly.

Forge 2.0 is our Jira app benchmark, with a budget of 300 model calls, twice Forge 1.0's. Goose, the open-source coding agent we build on, gets a Jira app built on Atlassian Forge and a long spec, and upgrades it with no help. A scorer then runs 105 checks against three seeded Jira sites. How it's built and scored is in the DeepSeek V4.1 Flash piece and in the Forge 2.0 scoring notes on the board.

The Forge 2.0 board on 17 October

This is the board as I read it on Saturday 17 October 2026, late morning EEST. The live board is the current word. It can move. Every model on it has exactly two independent builds, and the board ranks by the median of the two.

The column that matters most is the one people skip. A comparison group on our board is a set of builds made with the same verified task, scorer and goose build, among other things. Jev Router shares group 1 with exactly one other model, Gemini 4 Argon. Opus and Sol are in group 2. The board says, in its own words, "Compare models within the same comparison group." By our own rules, the first row and the third row aren't a fair pair.

What we found

1. Jev Router sent 117 of its 118 calls to GPT-6.1 Sol, verified one generation at a time through OpenRouter's API. The one exception went to GPT-6 Astra and cost 44.87% of its build's bill.
2. Both Jev builds beat both Sol builds on three of the 105 checks. No check went the other way for both Sol builds. Two of those three are checks where Sol's builds lost part of the marks, and on Forge 2.0 a partial miss costs as well as an outright failure.
3. Jev's lower build and Sol's higher build are near twins, 0.6398 and 0.6495. Under the rule our own comparison tool uses, that overlap alone is enough to call the pair unresolved.
4. On 23 of 117 routed calls, Sol thought for longer than Sol ever did when we called it directly, up to 3,327 reasoning tokens against a direct ceiling of 516. That's consistent with the router raising the effort, and I can't prove it.
5. The spread between two builds of one model is large on Forge 2.0. The median gap is 0.0832, Jev's is 0.2125, and Gemini 4 Argon's, in Jev's own comparison group, is 0.4983.

What is a model router, and what is Jev Router?

A model router is a model, or a policy, that sits in front of other models and decides which one answers each request. The pitch is cost. Send the easy request to a cheap model and the hard one to an expensive one, and you pay less for about the same quality. The RouteLLM paper from LMSYS puts it as routers "that dynamically select between a stronger and a weaker LLM during inference, aiming to optimize the balance between cost and response quality". Their blog post reported "cost reductions of over 85% on MT Bench" at 95% of GPT-4's performance. RouterBench starts from the premise that "no single model can optimally address all tasks and applications, particularly when balancing performance with cost." RouterEval found that a capable router can even "surpass the performance of the best single model in the pool", and that most existing methods "still have significant room for improvement".

Jev and Jev Router are two different things, and the search results mix them up.

Jev is TypeSafe's own model. TypeSafe launched it on 15 September 2026 in a post by its founder, Diogo Almeida, as its first "System One" model. In TypeSafe's docs, a System One model "evaluates a state and returns typed answers and probabilities", and the name comes from Kahneman's fast System 1 in Thinking, Fast and Slow. OpenRouter's Jev page answers the question I had directly: "Is Jev an LLM? No. Jev is a System One decision model. It returns typed answers, with probabilities associated with them. Jev doesn't return text, reasoning, or explanations."

Jev Router is the thing on our board. OpenRouter lists it as "TypeSafe: Jev Router", released on 25 September 2026, and its model page describes it like this:

"Jev Router picks the best model and reasoning effort for each request, balancing quality, speed, and cost. It runs on Jev, TypeSafe's first System One model, and adapts as your conversation evolves."

OpenRouter's Jev Router docs are more precise: "Jev makes the routing decision; the selected model generates the response." The router picks a model and an effort for each request, and "during supported agentic workflows, the router can retain a model, adjust its effort, or switch models as the task progresses." It has no price of its own. Each request is billed at the price of whichever model served it. I couldn't find a published list of the models it can pick from, and the docs say "the available models and routing policies can change over time." I couldn't find Jev Router mentioned anywhere on TypeSafe's own site. It's documented on OpenRouter, and I couldn't find either side saying who wrote the routing policy. At launch, crypto-news aggregators reported that OpenRouter's internal tests had Jev Router solving 237 of 423 agent tasks to the Auto Router's 130. I couldn't find that on OpenRouter's own pages, so I'm not leaning on it.

One more thing from TypeSafe that I'd rather quote than paraphrase. Their September post on benchmarks, "Lies, Damned Lies, and Benchmarks", tells readers to "run your own private evals, treat public ones with a grain of salt, and don't amplify every number or plot you see." I agree. That goes for our board too.

Jev Router vs GPT-6.1 Sol: 117 of 118 calls went to Sol

Each call goose makes through OpenRouter gets a generation id. Our build trees keep them. We asked OpenRouter's /api/v1/generation endpoint about every id from both Jev builds. Each record carries a model field (the model that served the call) and a router field.

The Gemini 2.5 Flash call in each build belongs to goose, our agent. It's goose's own small helper model, which ran once in each build with no router, and Sol's own builds have one as well. Take it out and the router handled 75 calls in build 1 and 43 in build 2. Sol calls, routed or direct, all hit the same upstream OpenAI endpoint on OpenRouter.

I'd only claim this for our task. OpenRouter's model page has a "Top models used by Jev Router" table for all of its traffic. When I read it on 17 October, DeepSeek V4.1 Flash led with about 36% of the tokens and GPT-6.1 Sol was second at about 18%. The page doesn't say over what window. On a long Atlassian Forge build with a 1-million-token context, the router chose Sol on 117 of 118 calls. Nothing here says it would decide that on your work.

The one call that went to GPT-6 Astra cost 44.87% of the build

Call 47 of build 1 went to GPT-6 Astra, OpenAI's flagship, which OpenRouter lists at five times Sol's price. It carried a 142,694-token prompt with 0 tokens cached, wrote 408 tokens back, and cost $1.8040675. The 74 Sol calls in the same build cost $2.2157321 together, so one call was 44.87% of a $4.02 bill. The Sol calls just before and after it cost $0.0526 and $0.0416.

The arithmetic matches Astra's cache-write price applied to the whole prompt. I read it as a cache write. The record doesn't say why the router picked Astra for that one call. OpenRouter itself warns about this cost in its router benchmark post: "Switching between models means requests get more expensive while the input cache is rebuilt." Our agent resends the whole conversation on each call, and 96% of each Jev build's input was a cache hit. One switch threw that away for one call. I'd have missed it without the per-call records.

What the router did to Sol's thinking, as far as we can see

Goose sent reasoning: {effort: "medium"} on every request we have logs for, in all four builds. OpenRouter's docs are clear that with Jev Router that's a suggestion: "a hint is not a guarantee of the setting sent to the final model."

The generation record has 45 fields. We checked them all and the effort isn't there. It does carry native_tokens_reasoning, and that shows something I can't prove. Called directly, Sol never used more than 516 reasoning tokens on a call, across 95 calls in two builds. Routed through Jev, it went over 516 on 19 of 74 Sol calls in build 1, up to 3,327, and on 4 of 43 in build 2. That's consistent with the router raising the effort on some calls. It's also consistent with other things I haven't ruled out.

The effort was knowable. OpenRouter documents an X-OpenRouter-Metadata: enabled request header that returns, for each request, the candidate models with their effort and the router's reason. Goose doesn't send it, so we have nothing. If you run a router through your own agent, send it.

The same docs say that "when the policy permits it, an expert advisor can assist the model doing the work." No generation record names a second model on any Sol call, and routed Sol calls billed at the same rate per token as Sol called directly, so I see no sign of one. Without the metadata header I can't rule it out.

Where Jev's builds beat Sol's, check by check

A Forge 2.0 score is two numbers multiplied. "Tests earned" is the weighted average of 105 checks. Reliability then multiplies it down: each of 19 groups of tests costs 1 − 0.10 × the shortfall of its worst test, so a group whose worst test fails completely costs ×0.90. The run pages show it step by step.

Look at tests earned first: 0.9409 and 0.9856 for Jev, 0.9498 and 0.9383 for Sol. On the average check the two are within a few hundredths. The gap opens in reliability, 0.68 and 0.8647 against 0.6838 and 0.5724.

Jev's build 1 and Sol's build 1 are the same story. Each had three groups with a check at zero, each lost a little more in a few others, and they finished 0.0097 apart. The other two builds are where the medians part. Jev's build 2 had one check at zero and only small misses elsewhere. Sol's build 2 had two zeros and then partial misses in seven more groups, half marks on two of them and about a quarter on three, which together cost it more than its two zeros did.

Three checks went to Jev in both builds and to neither Sol build:

Two of Jev's three wins are checks where Sol's builds lost part of the marks, 0.8333 and 0.5. There's no check where both Sol builds beat both Jev builds. Sol's build 2 edged both Jev builds on two checks, by small margins: event economy and the admin panel's labels. Sol's build 1 lost u_llm_explain outright. That single zero cost it ×0.90.

Then there's the other comparison, Jev against itself. Its two builds disagreed on 12 of 105 checks, Sol's two on 22. Build 1 answered "5/5 active sprints" exactly on the Rovo action check and build 2 answered "0/5". Build 1 hit 47 storage limit errors on one seeded site and build 2 none. Build 1 refused 1 of 958 person-facing requests for quota and build 2 refused 0 of 984. All three of build 1's zeros came from one of the three seeded Jira sites, the one where it scored 0.6399. Each check keeps its lowest result across the three sites (the excellence measurements take the mean), so one bad site can drag a whole build down.

Why a two-build agentic coding benchmark can't separate a router from its model

Jev's builds won a few checks where Sol's lost marks, and the scorer's reliability multiplier makes a handful of checks count for a lot. Whether that's the router helping, the newer goose, or a coin landing differently on a 20 to 45 minute build, I don't think two builds each can say.

The ranges overlap

Jev's lower build, 0.6398, is 0.0097 below Sol's higher build, 0.6495. Our comparison tool, forge2_compare.py, only names a winner between two models when every build of one beats every build of the other. Otherwise it prints:

"Unresolved comparison: insufficient independent builds or overlapping episode ranges."

To be exact: the report our benchmark session produced ran that tool on Jev's two builds alone, so its "unresolved" there means one model with no partner. Fed Jev and Sol together, the tool doesn't get as far as the ranges. It refuses, with "Cannot compare mixed task, engine, settings or scoring fixtures", because the goose build and the settings record differ. Even if they matched, 0.6398 against 0.6495 would fail its rule.

One model's two builds can be 0.5 apart

Across the 26 models with two Forge 2.0 builds, the median gap between a model's builds is 0.0832. Jev's is 0.2125, sixth largest. Gemini 4 Argon, in the same comparison group as Jev and on the same goose build, though called directly on Google rather than through OpenRouter, scored 0.7926 and then 0.2943 an hour and a quarter later. Its first build scored 0.9982, 0.7904 and 0.9981 on the three seeded sites, and its second 0.3725, 0.3753 and 0.3671. Same model, same comparison group, a different app. Every Forge 2.0 pair is worth reading with that in mind.

That isn't peculiar to us. "On Randomness in Agentic Evals" collected 60,000 trajectories on SWE-bench Verified and found single-run scores "vary by 2.2 to 6.0 percentage points depending on which run is selected", with trajectories that "diverge early, often within the first few percent of tokens". Anthropic's statistical approach to model evaluations makes the general point that an eval with few questions has wide intervals, so "models will need to have a large underlying difference in capabilities in order to register a statistically significant result". Forge 2.0 is one long build scored on three seeded sites, which is very few samples per model.

The engines differ

Jev's builds ran this morning on goose 3.0.116. Sol's ran overnight on 11 to 12 October on a 3.0.110 build. The two cohorts differ only in the engine binary and its settings record. Everything else in the cohort record matches, down to the seeds and the effort we asked for. We checked what the model actually saw: the first request of each build had the same five tools and the same 2,621-character system prompt, and the task text differs only in the timestamp, the working directory and the context-window line. None of the four builds hit a retry or a rate-limit wait.

There are 73 commits between those two goose versions, 13 of them in the engine. Two touch the OpenRouter path. One changes when goose sends a reasoning effort to a model it doesn't recognise, the other how it waits out a rate limit that arrives mid-stream. Our logs show effort: medium going out on all four builds, and no build hit a rate limit, so I don't think either made the requests differ. I'm inferring that from the logs. The board still puts the two in different groups, and it should.

Epoch AI found that changing the scaffold alone moved SWE-bench Verified by up to 11% for GPT-5 and 15% for Kimi K2 Thinking. Our change between versions is far smaller than swapping scaffolds. It's still a change.

The router may have chosen a different effort

If the router raised the effort on hard steps, the "Jev" builds weren't Sol at medium. We asked for medium in Sol's own pair. That could be a real benefit of the router. We didn't record the effort, so it stays a hypothesis.

What it cost

Claude Opus 5.5's two builds billed $42.50 for a 0.6745 median. Jev Router's billed $5.17 for 0.746, and Sol's own $2.38 for 0.5933.

Take out the Astra call and build 1's 74 Sol calls came to $2.22, or $0.0299 a call, against $0.0245 and $0.0256 for Sol called directly. Jev's build 2 cost almost exactly what Sol's build 1 cost. The router has no fee of its own. It does carry the chance of a switch, and here one switch cost 81% as much as the 74 Sol calls in the same build.

The Jev run pages don't show a bill. Those figures come from OpenRouter's generation records, and the build's own cost file marks both bills complete. Sol's run pages show theirs.

What could be wrong with my reading

1. Two builds per model is a small sample. Everything above describes four builds. I wouldn't defend it as a ranking.
2. The engines differ, as described. I checked that the model saw the same tools and prompt. I didn't measure what 73 commits do to a score.
3. Our request logs keep only the last 10 requests per build, so "goose sent medium" is verified for those 10 in each build.
4. Our own run pages title Jev's builds "typesafe/jev-router, single model via OpenRouter". That label is ours, and it's wrong for a router.
5. Jev Router's earlier results don't settle it. On Gauntlet 7.2, our payments-app benchmark, its single run scored 0.6975 against Sol's 0.7978. Both are score caps, 0.699 for Jev after one failed check and 0.799 for Sol. Their raw scores were 0.9706 and 0.9768. I don't have that run's generation ids, so I don't know which models it routed to there.

How to check which model Jev Router actually ran

If you're about to pick a router because of where it sits on a leaderboard, ours included, check what it ran first. The API calls below are from OpenRouter's docs. Step 3 is my advice.

1. Keep the generation id of each call. OpenRouter returns it on each response.
2. Ask GET https://openrouter.ai/api/v1/generation?id=<id> for each one, with your API key. Read model and router first, then total_cost. We did this for all 120 generation ids from Jev's two builds, the 118 routed calls plus goose's two helper calls.
3. Count the models. If one model served nearly every call, compare the router's result with that model's own result on the same harness, same version, same week.
4. To see the reasoning effort, send X-OpenRouter-Metadata: enabled on the request. The router's metadata then lists the candidates with their effort and the reason for the pick.

Then look at the spread. Jev's lead over Sol's median is 0.1527, and Jev's own two builds are 0.2125 apart. When a router's lead is smaller than the gap between its own builds, I wouldn't trust the order, however the medians sort.

So why is Jev number one?

Because its two builds of GPT-6.1 Sol had a better median than Sol's own two, which ran five days earlier on an older goose, and the router was free to change Sol's effort. Its first build finished level with Sol's first. Its second build had a single check at zero and small misses elsewhere, while Sol's second lost marks across nine groups. I can't say from these records whether the router's effort choices helped.

I can say something narrower, and I think more useful. On a long agentic build, Jev Router picked GPT-6.1 Sol on 117 of 118 calls, and paid more per call than Sol called directly, with one expensive switch as the biggest single piece. If you want GPT-6.1 Sol for this kind of work, calling Sol directly gets you the same model without the switch risk. You give up the router's effort choices. I don't know what those were worth. If you want a router for mixed traffic, where its own usage table puts DeepSeek V4.1 Flash first, at about a third of its tokens, this benchmark doesn't test that at all.

Goose built all four apps, and the next thing it builds should be a third and fourth Jev build on 3.0.116 next to a fresh Sol pair on the same engine, with the metadata header on. Until then, the Forge 2.0 board links the four run pages: Jev build 1, Jev build 2, Sol build 1 and Sol build 2, with all their checks and recordings. Argon's pair, 0.7926 and 0.2943, is the best argument for reading all of them with the spread in mind. For how GPT-6.1 Sol did on our earlier boards, see Claude Sonnet 5.5 vs Opus 5.5 vs GPT-6.1 Sol, and for small against big inside one family, Qwen3.8 Flash vs Max Prime.