Two independent Jira app builds

Forge 2.0 Benchmark

See which AI builds more for less. Run the same test in Goose.

Models upgrade the same Jira app in two independent builds, each tested and scored, with recorded inference spend shown for both.

Forge leaderboard

Forge results of one version, ranked together. Not comparable with the Gauntlet board: a different task and a different scorer.

Hover a row or use its info button for build details and recorded spend.

Cloud baselines show the median and range of independent builds within the same verified task, engine and captured comparison cohort. The reasoning policy requests medium effort where supported, or enables advertised reasoning at provider defaults when medium cannot be selected or reasoning metadata is absent, subject to each route's native qualification. Each model's literal request is recorded; effective reasoning is not assumed identical across providers. One build is provisional. Each build keeps its own score and evidence. Compare models within the same comparison group.

Model / runScore / median
1. Opus 5.5anthropic/claude-opus-5.52 independent buildsComparison group 1Range 0.6196–0.7294Build 1: 0.7294Build 2: 0.6196
0.6745
2. GPT 6.1 Solopenai/gpt-6.1-sol2 independent buildsComparison group 1Range 0.5371–0.6495Build 1: 0.6495Build 2: 0.5371
0.5933
3. qwen3.8-max-primeqwen/qwen3.8-max-prime2 independent buildsComparison group 2Range 0.5015–0.6515Build 1: 0.6515Build 2: 0.5015
0.5765
4. Sonnet 5.5anthropic/claude-sonnet-5.52 independent buildsComparison group 2Range 0.5485–0.5899Build 1: 0.5485Build 2: 0.5899
0.5692
5. MiMo v2.6 Flashxiaomi/mimo-v2.6-flash2 independent buildsComparison group 1Range 0.5028–0.5538Build 1: 0.5538Build 2: 0.5028
0.5283
6. ember-1fireworks/ember-12 independent buildsComparison group 2Range 0.4710–0.5312Build 1: 0.4710Build 2: 0.5312
0.5011
7. glm-5.3-primez-ai/glm-5.3-prime2 independent buildsComparison group 2Range 0.4046–0.4557Build 1: 0.4046Build 2: 0.4557
0.4302
8. Gemini 3.8 Flashgoogle/gemini-3.8-flash2 independent buildsComparison group 2Range 0.2999–0.4958Build 1: 0.4958Build 2: 0.2999
0.3979
9. Qwen 3.8 Omni Flashqwen/qwen3.8-omni-flash2 independent buildsComparison group 2Range 0.1659–0.5104Build 1: 0.5104Build 2: 0.1659
0.3381
10. DeepSeek 4.1 Flashdeepseek/deepseek-v4.1-flash2 independent buildsComparison group 1Range 0.2893–0.3790Build 1: 0.2893Build 2: 0.3790
0.3342
11. muse-wondermeta/muse-wonder2 independent buildsComparison group 3Range 0.2366–0.4203Build 1: 0.4203Build 2: 0.2366
0.3285
12. grok-4.7x-ai/grok-4.72 independent buildsComparison group 2Range 0.1815–0.4682Build 1: 0.4682Build 2: 0.1815
0.3248
13. muse-spark-1.3meta/muse-spark-1.32 independent buildsComparison group 2Range 0.2202–0.2968Build 1: 0.2202Build 2: 0.2968
0.2585
14. step-5-previewstepfun/step-5-preview2 independent buildsComparison group 2Range 0.2356–0.2749Build 1: 0.2749Build 2: 0.2356
0.2552
15. GPT 6 Lunaopenai/gpt-6-luna2 independent buildsComparison group 2Range 0.2197–0.2580Build 1: 0.2197Build 2: 0.2580
0.2389
16. hy4-previewtencent/hy4-preview2 independent buildsComparison group 2Range 0.0555–0.3901Build 1: 0.0555Build 2: 0.3901
0.2228
17. qwen3.8-27bqwen/qwen3.8-27b2 independent buildsComparison group 4Range 0.0617–0.3541Build 1: 0.3541Build 2: 0.0617
0.2079
18. glm-5.3-flashxz-ai/glm-5.3-flashx2 independent buildsComparison group 2Range 0.0714–0.3337Build 1: 0.3337Build 2: 0.0714
0.2026
19. ling-3.1-flashinclusionai/ling-3.1-flash2 independent buildsComparison group 2Range 0.1582–0.1930Build 1: 0.1930Build 2: 0.1582
0.1756
20. GLM 5.3z-ai/glm-5.32 independent buildsComparison group 2Range 0.1453–0.1850Build 1: 0.1850Build 2: 0.1453
0.1652
21. claude-haiku-5.5anthropic/claude-haiku-5.52 independent buildsComparison group 2Range 0.0610–0.0632Build 1: 0.0632Build 2: 0.0610
0.0621
22. Qwen 3.8 27Bqwen/qwen3.8-27b2 independent buildsComparison group 2Range 0.0014–0.0617Build 1: 0.0014Build 2: 0.0617
0.0316
23. GLM 5.3 Flashz-ai/glm-5.3-flash2 independent buildsComparison group 2Range 0.0014–0.0617Build 1: 0.0014Build 2: 0.0617
0.0316
Community runCloud baselineForge 2.0 · 23 entries · 23 baselines
Invalidated Forge 2.0 run archive

Previous results are excluded from the ranking. Their original evidence remains available below.

About the Forge benchmark

Forge 1.0, frozen