Cloud baseline

Qwen4 Max, single model via OpenRouter

qwen/qwen4-max · scorer forge-2.0

qwen/qwen4-max · single agentNov 16, 2026
Overall
0.3473
Benchmark
Forge 2.0
Model build
71m 18s

Recorded inference cost

$3.67

From the saved OpenRouter generation billing summary (144 recorded requests). Whether auxiliary model calls were included was not retained. Scoring reuses the saved build; this amount describes the original inference spend.

Token usage

Tokens: 30M in · 181k out · 29.5M cached (exact, OpenRouter records)

30,166,539 tokens over 144 model calls: every call of the run, the model's own and the harness's helper calls, counted from OpenRouter records. Reasoning tokens as the provider reports them: 104,054 (not added to the total).

  • qwen/qwen4-max

    In 30M · Out 181k · Cached 29.5M · Reasoning 104k · 143 calls

  • google/gemini-2.5-flash

    In 1.4k · Out 6 · Cached 0 · Reasoning 0 · 1 call

Tier breakdown

Each bar is the tier's mean, from 0 to 1 (for E, its mean × the excellence gate). The percentage is the tier's share of the score. Mean × share, added over all tiers, is the tests earned figure below.

L · Lint and bundles1.001.06%
K · Platform currency0.9251.32%
T · Event pipeline0.97882.11%
R · Reconcile0.99611.85%
S · Storage0.751.06%
B · Resolvers and permissions0.61111.58%
U · UI function0.09552.11%
V · Visual1.001.06%
A · Rovo0.95831.06%
E · Excellence0.70081.8%
R1 · Migration1.0012%
R2 · Dosing0.678613%
R3 · Time limits1.008%
R4 · Changing world0.8010%
R5 · Admin panel1.0010%
R6 · CI web trigger1.008%
R7 · Custom field0.99515%
R8 · Forge LLM0.83334%
R9 · Boot budget0.66675%
R10 · Native JQL0.78710%

Scoring detail

How this score was built

Per-test scorer output, as posted by the app. Expand a section to inspect its tests.

Correctness combines the lowest score for each test across the scoring fixtures; economy measurements use their mean. The combined result can therefore be lower than every individual fixture’s final score. Critical defects reduce the weighted result, followed by the reliability multiplier, before any score cap applies.

The rules behind these numbers: Forge 2.0 requirements, weights and score caps.

The score, step by step

0.8581 tests earned × 1.0000 critical defects × 0.4048 reliability = 0.3473 final score

tests earned = 0.15 × 0.7651 v1 + 0.7433 v2 (out of 0.85)

v1 = 0.88 × 0.7738 tiers L–A + 0.12 × 0.8123 gate × 0.8626 excellence

Reliability: every tier in the breakdown above, except Excellence, is a group of tests. Each group of tests multiplies the score by 1 − 0.10 × the shortfall of its worst test: by 0.90 when that test fails completely, by 0.95 when it scores 0.5.

  • K Platform currency: worst test k_widget_edit_bridge, score 0.25 × 0.9250
  • T Event pipeline: worst test t_reestimate_followed, score 0.8333 × 0.9833

    Already counted in this group: t_event_rows

  • R Reconcile: worst test r_pagination, score 0.9692 × 0.9969
  • S Storage: worst test s_index_order, score 0 × 0.9000
  • B Resolvers and permissions: worst test b_comment_adf_as_user, score 0 × 0.9000

    Already counted in this group: b_hidden_count, b_comment_exactly_once

  • U UI function: worst test u_ledger_table, score 0 × 0.9000

    Already counted in this group: u_widget_numbers, u_widget_chart, u_widget_edit_config, u_ledger_sort, u_issue_router, u_comment_flow, u_modal_close, u_not_started, u_widget_live, u_llm_explain

  • A Rovo: worst test a_action_permissions, score 0.8333 × 0.9833
  • R2 Dosing: worst test r2_person_never_quota_refused, score 0 × 0.9000

    Already counted in this group: r2_rate_limit_reaction

  • R4 Changing world: worst test r4_sprint_close_final, score 0 × 0.9000
  • R7 Custom field: worst test r7_values_fresh, score 0.9903 × 0.9990
  • R8 Forge LLM: worst test r8_tool_call_scope, score 0.5 × 0.9500
  • R9 Boot budget: worst test r9_sprint_boot, score 0 × 0.9000
  • R10 Native JQL: worst test r10_empty_and_closed, score 0 × 0.9000

    Already counted in this group: r10_precomputation_refresh, r10_shared_permissions

v1 is this run's score on the tests carried over from Forge 1.0: 15% of the total. v2 is the requirements R1–R10, each at its share of the score: the other 85%. Critical defects multiply the whole score, and so does each group of tests, by 1 − 0.10 × the shortfall of its worst test.

Excellence gate 0.8123: the average of the 18 conditions below, each counted at the value shown. Green is met in full; red is not, and lowers the gate. The 3 excellence tests are paid in proportion to the gate; together they are 1.8% of the total.

t_event_rows 0.9971r_heal_dropped 1.00u_widget_numbers 0.375u_ledger_table 0.00b_comment_adf_as_user 0.00k_consumer_shape 1.00k_current_apis 1.00k_dashboard_widget 1.00k_entity_declared 1.00k_llm_model_current 1.00k_manifest_semantics 1.00k_rovo_mcp 1.00k_rovo_skill 1.00k_runtime_risks 1.00k_widget_edit_bridge 0.25v_console_clean 1.00v_csp_clean 1.00l_lint_warnings 1.00

Earned score 0.3473 · No cap applies · Final score 0.3473

Under a cap the final score is min(earned, cap − 0.05 × (1 − earned)): always below the cap, and closer to it the more the run earned. With no cap it is the earned score. Avoiding a cap adds no points.

The dashboard widget's numbers and chart, its board choice saved through the host, live updates through Forge Realtime, the sprint action's ledger table, sorting, issue links, comment flow, the Forge LLM explanation, close and not-started states.

u widget loadsdiagnostic, no weight

1.00

4/8 configured widget views render a sprint with its numbers

u widget numbers

0.375

worst of 3 scoring sites (seed d031c3ee22136508): widget numbers: 3.00/8 views exact (sprints by startDate, four §1 numbers each)

u widget chart

0.375

worst of 3 scoring sites (seed d031c3ee22136508): 3/8 charts on one linear scale within 1px; ['no single svg[data-testid=chart]', 'no single svg[data-testid=chart]']

u widget edit config

0.30

worst of 3 scoring sites (seed d031c3ee22136508): edit/config 3/10: pick0_reopen_pressed, pick1_view_shows_board, pick1_reopen_pressed, pick2_view_shows_board, pick2_reopen_pressed, pick3_reopen_pressed, second_instance_

u ledger table

0.00

worst of 3 scoring sites (seed d031c3ee22136508): ledger tables: 0.00/6 exact (rows, cells, default order)

u ledger sort

0.00

worst of 3 scoring sites (seed d031c3ee22136508): vacuous — precondition unmet: table rows rendered

u issue router

0.00

worst of 3 scoring sites (seed d031c3ee22136508): vacuous — precondition unmet: table rows rendered

u comment flow

0.00

worst of 3 scoring sites (seed d031c3ee22136508): required app surface absent: post-summary flow

u modal close

0.00

worst of 3 scoring sites (seed d031c3ee22136508): vacuous — precondition unmet: modal rendered

u not started

0.00

worst of 3 scoring sites (seed d031c3ee22136508): future sprint context: not-started not shown alone

u widget live

0.00

worst of 3 scoring sites (seed 3c04f61f28a4a511): moved sprints showing the new numbers live 0/1; gate failed: subscribed; 0 idle invoke(s)

u llm explain

0.00

worst of 3 scoring sites (seed d031c3ee22136508): vacuous — precondition unmet: explain control rendered

Graded browser recording

Loading duration…

Watch the full graded browser recording at its original speed: the dashboard widget, the sprint action, the not-started sprint and the Forge LLM cases, as the scorer drove them on the run's own seed, the first of the three scoring sites. The test results and screenshots on this page provide the wider evidence. Pause or scrub to inspect a view.

The UI Kit admin panel is not in the recording: it is graded from its component tree.

Ready to play0:00 / —:—
Open recording file

Drag the timeline to seek. Keyboard: arrow keys move 5 seconds; Home and End jump to the start and end.

Full graded browser recording: dashboard widget (before configuration, edit and Save, light and dark at 380 px, a second instance at 1180 px), sprint action (light and dark, sort, issue link, select, comment post, double click, forbidden post, explain, close), the not-started sprint and the Forge LLM cases, where the app has each. The UI Kit admin panel is not in the recording.

Screenshots

Pictures from one of the three scoring sites, the one on the run's own seed. The dashboard widget and the sprint action are captured from the built app in a browser during grading. The admin panel picture is drawn by the benchmark from the app's component tree, not by Jira, and nothing is graded from that picture. A test keeps the score of its worst site (the excellence measurements keep their mean), so a test can fail on a site these pictures do not show. A detail that opens "worst of …" names the seed of the site the test scored worst on, which can be the pictured one.

1Dashboard widget · light
2Dashboard widget · dark
3Sprint action · light
4Sprint action · dark
5Admin panel (UI Kit) · light: drawn by the benchmark from the app's component tree, not by Jira.

Run details

Model
qwen/qwen4-max
Engine events
0
Repair rounds
0
Started
Nov 16, 2026, 12:07 PM
Finished
Nov 16, 2026, 01:57 PM
Call budget
With wrap-up nudge, version 2: as the call budget ran low, the agent harness told the model to start no new work and finish

How the benchmark is scored