Two independent Jira app builds

Forge 2.0 Benchmark

See which AI builds more for less. Run the same test in Goose.

Models upgrade the same Jira app in two independent builds, each tested and scored, with recorded inference spend shown for both.

Forge leaderboard

Forge results of one version, ranked together. Not comparable with the Gauntlet board: a different task and a different scorer.

Hover a row or use its info button for build details and recorded spend.

Cloud baselines show the median and range of independent builds within the same verified task, engine and captured comparison cohort. The reasoning policy requests medium effort where supported, or enables advertised reasoning at provider defaults when medium cannot be selected or reasoning metadata is absent, subject to each route's native qualification. Each model's literal request is recorded; effective reasoning is not assumed identical across providers. One build is provisional. Each build keeps its own score and evidence. Compare models within the same comparison group.

Model / runScore / median
1. jev-routertypesafe/jev-router2 independent buildsComparison group 1Range 0.6398–0.8523Build 1: 0.6398Build 2: 0.8523
0.7460
2. Opus 5.5anthropic/claude-opus-5.52 independent buildsComparison group 2Range 0.6196–0.7294Build 1: 0.7294Build 2: 0.6196
0.6745
3. GPT 6.1 Solopenai/gpt-6.1-sol2 independent buildsComparison group 2Range 0.5371–0.6495Build 1: 0.6495Build 2: 0.5371
0.5933
4. qwen3.8-max-primeqwen/qwen3.8-max-prime2 independent buildsComparison group 3Range 0.5015–0.6515Build 1: 0.6515Build 2: 0.5015
0.5765
5. Sonnet 5.5anthropic/claude-sonnet-5.52 independent buildsComparison group 3Range 0.5485–0.5899Build 1: 0.5485Build 2: 0.5899
0.5692
6. gemini-4-argongemini-4-argon2 independent buildsComparison group 1Range 0.2943–0.7926Build 1: 0.7926Build 2: 0.2943
0.5434
7. MiMo v2.6 Flashxiaomi/mimo-v2.6-flash2 independent buildsComparison group 2Range 0.5028–0.5538Build 1: 0.5538Build 2: 0.5028
0.5283
8. ember-1fireworks/ember-12 independent buildsComparison group 3Range 0.4710–0.5312Build 1: 0.4710Build 2: 0.5312
0.5011
9. glm-5.3-primez-ai/glm-5.3-prime2 independent buildsComparison group 3Range 0.4046–0.4557Build 1: 0.4046Build 2: 0.4557
0.4302
10. qwen3.8-flashqwen/qwen3.8-flash2 independent buildsComparison group 4Range 0.3164–0.4990Build 1: 0.3164Build 2: 0.4990
0.4077
11. Gemini 3.8 Flashgoogle/gemini-3.8-flash2 independent buildsComparison group 3Range 0.2999–0.4958Build 1: 0.4958Build 2: 0.2999
0.3979
12. Qwen 3.8 Omni Flashqwen/qwen3.8-omni-flash2 independent buildsComparison group 3Range 0.1659–0.5104Build 1: 0.5104Build 2: 0.1659
0.3381
13. DeepSeek 4.1 Flashdeepseek/deepseek-v4.1-flash2 independent buildsComparison group 2Range 0.2893–0.3790Build 1: 0.2893Build 2: 0.3790
0.3342
14. muse-wondermeta/muse-wonder2 independent buildsComparison group 5Range 0.2366–0.4203Build 1: 0.4203Build 2: 0.2366
0.3285
15. grok-4.7x-ai/grok-4.72 independent buildsComparison group 3Range 0.1815–0.4682Build 1: 0.4682Build 2: 0.1815
0.3248
16. muse-spark-1.3meta/muse-spark-1.32 independent buildsComparison group 3Range 0.2202–0.2968Build 1: 0.2202Build 2: 0.2968
0.2585
17. step-5-previewstepfun/step-5-preview2 independent buildsComparison group 3Range 0.2356–0.2749Build 1: 0.2749Build 2: 0.2356
0.2552
18. GPT 6 Lunaopenai/gpt-6-luna2 independent buildsComparison group 3Range 0.2197–0.2580Build 1: 0.2197Build 2: 0.2580
0.2389
19. hy4-previewtencent/hy4-preview2 independent buildsComparison group 3Range 0.0555–0.3901Build 1: 0.0555Build 2: 0.3901
0.2228
20. glm-5.3-flashxz-ai/glm-5.3-flashx2 independent buildsComparison group 3Range 0.0714–0.3337Build 1: 0.3337Build 2: 0.0714
0.2026
21. ling-3.1-flashinclusionai/ling-3.1-flash2 independent buildsComparison group 3Range 0.1582–0.1930Build 1: 0.1930Build 2: 0.1582
0.1756
22. GLM 5.3z-ai/glm-5.32 independent buildsComparison group 3Range 0.1453–0.1850Build 1: 0.1850Build 2: 0.1453
0.1652
23. Qwen3.8-27B 8-bit · local Goose · single modelMihai-LeanZero/Qwen3.8-27B-Atlassian-Q8-base-mlxBuild 1: 0.1457
0.1457
24. Qwen3.8-27B 8-bit · local Goose · single modelMihai-LeanZero/Qwen3.8-27B-Atlassian-Q8-base-mlxBuild 1: 0.1342
0.1342
25. claude-haiku-5.5anthropic/claude-haiku-5.52 independent buildsComparison group 3Range 0.0610–0.0632Build 1: 0.0632Build 2: 0.0610
0.0621
26. Qwen 3.8 27Bqwen/qwen3.8-27b2 independent buildsComparison group 3Range 0.0014–0.0617Build 1: 0.0014Build 2: 0.0617
0.0316
27. GLM 5.3 Flashz-ai/glm-5.3-flash2 independent buildsComparison group 3Range 0.0014–0.0617Build 1: 0.0014Build 2: 0.0617
0.0316
Community runCloud baselineForge 2.0 · 27 entries · 25 baselines
Invalidated Forge 2.0 run archive

Previous results are excluded from the ranking. Their original evidence remains available below.

About the Forge benchmark

Forge 1.0, frozen