Two independent Jira app builds

Forge 2.0 benchmark

See which AI builds more for less, graded by running what it ships.

Forge leaderboard

Forge results of one version, ranked together. Not comparable with the Gauntlet board: a different task and a different scorer.

Models upgrade the same Jira app in two independent builds, each tested and scored, with recorded inference spend shown for both.

Hover a row or use its info button for build details and recorded spend."With wrap-up nudge" means the agent harness told the model to start no new work and finish as its call budget ran low.

Model / runScore / median
1. Jev Routertypesafe/jev-router2 independent buildsComparison group 1Range 0.6398–0.8523Build 1: 0.6398Build 2: 0.8523
0.7460
2. Opus 5.5anthropic/claude-opus-5.52 independent buildsComparison group 2Range 0.6196–0.7294Build 1: 0.7294Build 2: 0.6196
0.6745
3. Gemini 4 Carbongoogle/gemini-4-carbon2 independent buildsComparison group 3Range 0.5425–0.7862with wrap-up nudgeBuild 1: 0.5425Build 2: 0.7862
0.6643
4. Particle-1particle-ai/particle-12 independent buildsComparison group 4Range 0.5717–0.6530with wrap-up nudgeBuild 1: 0.5717Build 2: 0.6530
0.6123
5. GPT 6.1 Solopenai/gpt-6.1-sol2 independent buildsComparison group 2Range 0.5371–0.6495Build 1: 0.6495Build 2: 0.5371
0.5933
6. Qwen3.8 Max Primeqwen/qwen3.8-max-prime2 independent buildsComparison group 5Range 0.5015–0.6515Build 1: 0.6515Build 2: 0.5015
0.5765
7. Sonnet 5.5anthropic/claude-sonnet-5.52 independent buildsComparison group 5Range 0.5485–0.5899Build 1: 0.5485Build 2: 0.5899
0.5692
8. Gemini 4 Argongemini-4-argon2 independent buildsComparison group 1Range 0.2943–0.7926Build 1: 0.7926Build 2: 0.2943
0.5434
9. MiMo v2.6 Flashxiaomi/mimo-v2.6-flash2 independent buildsComparison group 2Range 0.5028–0.5538Build 1: 0.5538Build 2: 0.5028
0.5283
10. Hy4tencent/hy42 independent buildsComparison group 6Range 0.4532–0.5993with wrap-up nudgeBuild 1: 0.5993Build 2: 0.4532
0.5262
11. Ember-1fireworks/ember-12 independent buildsComparison group 5Range 0.4710–0.5312Build 1: 0.4710Build 2: 0.5312
0.5011
12. Hy4 previewtencent/hy4-preview2 independent buildsComparison group 7Range 0.4540–0.5165Build 1: 0.5165Build 2: 0.4540
0.4852
13. ERNIE-Light-DT-1.0baidu/ernie-light-dt-1.0-intl2 independent buildsComparison group 8Range 0.4675–0.4847with wrap-up nudgeBuild 1: 0.4675Build 2: 0.4847
0.4761
14. Gemini 4 Hydrogengoogle/gemini-4-hydrogen2 independent buildsComparison group 3Range 0.4389–0.5032with wrap-up nudgeBuild 1: 0.5032Build 2: 0.4389
0.4710
15. DeepSeek 4.1 Flash 1017deepseek/deepseek-v4.1-flash-10172 independent buildsComparison group 9Range 0.2422–0.6618Build 1: 0.2422Build 2: 0.6618
0.4520
16. GLM 5.3 Primez-ai/glm-5.3-prime2 independent buildsComparison group 5Range 0.4046–0.4557Build 1: 0.4046Build 2: 0.4557
0.4302
17. GPT-6.1 Terraopenai/gpt-6.1-terra2 independent buildsComparison group 10Range 0.4174–0.4384Build 1: 0.4174Build 2: 0.4384
0.4279
18. Qwen3.8 Flashqwen/qwen3.8-flash2 independent buildsComparison group 11Range 0.3164–0.4990Build 1: 0.3164Build 2: 0.4990
0.4077
19. Gemini 3.8 Flashgoogle/gemini-3.8-flash2 independent buildsComparison group 5Range 0.2999–0.4958Build 1: 0.4958Build 2: 0.2999
0.3979
20. Qwen4 Maxqwen/qwen4-max2 independent buildsComparison group 8Range 0.3473–0.4087with wrap-up nudgeBuild 1: 0.3473Build 2: 0.4087
0.3780
21. MiniMax M3.1 Flashminimax/minimax-m3.1-flash:free2 independent buildsComparison group 12Range 0.2134–0.5366Build 1: 0.2134Build 2: 0.5366
0.3750
22. Kimi K3.1 Promoonshotai/kimi-k3.1-pro2 independent buildsComparison group 13Range 0.1532–0.5342Build 1: 0.5342Build 2: 0.1532
0.3437
23. Qwen 3.8 Omni Flashqwen/qwen3.8-omni-flash2 independent buildsComparison group 5Range 0.1659–0.5104Build 1: 0.5104Build 2: 0.1659
0.3381
24. DeepSeek 4.1 Flashdeepseek/deepseek-v4.1-flash2 independent buildsComparison group 2Range 0.2893–0.3790Build 1: 0.2893Build 2: 0.3790
0.3342
25. Muse Wondermeta/muse-wonder2 independent buildsComparison group 14Range 0.2366–0.4203Build 1: 0.4203Build 2: 0.2366
0.3285
26. Grok 4.7x-ai/grok-4.72 independent buildsComparison group 5Range 0.1815–0.4682Build 1: 0.4682Build 2: 0.1815
0.3248
27. Nex-N2.6nex-agi/nex-n2.62 independent buildsComparison group 3Range 0.1967–0.4322with wrap-up nudgeBuild 1: 0.4322Build 2: 0.1967
0.3145
28. Muse Embermeta/muse-ember2 independent buildsComparison group 6Range 0.2870–0.3232with wrap-up nudgeBuild 1: 0.2870Build 2: 0.3232
0.3051
29. Muse Spark 1.3meta/muse-spark-1.32 independent buildsComparison group 5Range 0.2202–0.2968Build 1: 0.2202Build 2: 0.2968
0.2585
30. Step 5 Previewstepfun/step-5-preview2 independent buildsComparison group 5Range 0.2356–0.2749Build 1: 0.2749Build 2: 0.2356
0.2552
31. GPT 6 Lunaopenai/gpt-6-luna2 independent buildsComparison group 5Range 0.2197–0.2580Build 1: 0.2197Build 2: 0.2580
0.2389
32. Mistral Large 4mistralai/mistral-large-4-02 independent buildsComparison group 9Range 0.1086–0.3304Build 1: 0.1086Build 2: 0.3304
0.2195
33. GLM 5.3 FlashXz-ai/glm-5.3-flashx2 independent buildsComparison group 5Range 0.0714–0.3337Build 1: 0.3337Build 2: 0.0714
0.2026
34. Qwen3.8-27B 8-bitMihai-LeanZero/Qwen3.8-27B-Atlassian-Q8-base-mlx2 independent buildsRange 0.1128–0.2576Build 1: 0.2576Build 2: 0.1128
0.1852
35. Ling 3.1 Flashinclusionai/ling-3.1-flash2 independent buildsComparison group 5Range 0.1582–0.1930Build 1: 0.1930Build 2: 0.1582
0.1756
36. Glimpseelorian/glimpse2 independent buildsComparison group 15Range 0.1595–0.1732Build 1: 0.1595Build 2: 0.1732
0.1663
37. GLM 5.3z-ai/glm-5.32 independent buildsComparison group 5Range 0.1453–0.1850Build 1: 0.1850Build 2: 0.1453
0.1652
38. Flint 40B-A2.5B 4-bitRecursiveSI/flint-int42 independent buildsRange 0.1073–0.1172Build 1: 0.1073Build 2: 0.1172
0.1123
39. Muse Spark Openmeta/muse-spark-open2 independent buildsComparison group 15Range 0.0877–0.0910Build 1: 0.0910Build 2: 0.0877
0.0893
40. Ripple 65B Previewricursiveai/ripple-65b-preview2 independent buildsComparison group 16Range 0.0770–0.0858with wrap-up nudgeBuild 1: 0.0858Build 2: 0.0770
0.0814
41. Beam 1.0 501B A23Breflection/beam-1.0-501b-a23b2 independent buildsComparison group 15Range 0.0624–0.0641Build 1: 0.0641Build 2: 0.0624
0.0633
42. Haiku 5.5anthropic/claude-haiku-5.52 independent buildsComparison group 5Range 0.0610–0.0632Build 1: 0.0632Build 2: 0.0610
0.0621
43. LFM3-1.7Bliquid/lfm-3-1.7b2 independent buildsComparison group 16Range 0.0617–0.0617with wrap-up nudgeBuild 1: 0.0617Build 2: 0.0617
0.0617
44. Qwen 3.8 27Bqwen/qwen3.8-27b2 independent buildsComparison group 5Range 0.0014–0.0617Build 1: 0.0014Build 2: 0.0617
0.0316
45. GLM 5.3 Flashz-ai/glm-5.3-flash2 independent buildsComparison group 5Range 0.0014–0.0617Build 1: 0.0014Build 2: 0.0617
0.0316
Community runCloud baselineForge 2.0 · 45 entries · 43 baselines

Cloud baselines show the median and range of independent builds within the same verified task, engine and captured comparison cohort. The reasoning policy requests medium effort where supported, or enables advertised reasoning at provider defaults when medium cannot be selected or reasoning metadata is absent, subject to each route's native qualification. Each model's literal request is recorded; effective reasoning is not assumed identical across providers. One build is provisional. Each build keeps its own score and evidence. Compare models within the same comparison group. A local or community entrant with two or more receipted builds shows the same median and range under its own title (green); it is not a cloud baseline.

Invalidated Forge 2.0 run archive

Previous results are excluded from the ranking. Their original evidence remains available below.

About the Forge benchmark

Forge 1.0, frozen