Cloud baseline

Qwen4 Max, single model via OpenRouter

qwen/qwen4-max · scorer forge-2.0

qwen/qwen4-max · single agentNov 16, 2026
Overall
0.4087
Benchmark
Forge 2.0
Model build
59m 36s

Recorded inference cost

$3.59

From the saved OpenRouter generation billing summary (177 recorded requests). Whether auxiliary model calls were included was not retained. Scoring reuses the saved build; this amount describes the original inference spend.

Token usage

Tokens: 28.8M in · 158k out · 28.2M cached (exact, OpenRouter records)

28,907,924 tokens over 177 model calls: every call of the run, the model's own and the harness's helper calls, counted from OpenRouter records. Reasoning tokens as the provider reports them: 91,539 (not added to the total).

  • qwen/qwen4-max

    In 28.7M · Out 158k · Cached 28.2M · Reasoning 91.5k · 176 calls

  • google/gemini-2.5-flash

    In 1.4k · Out 3 · Cached 0 · Reasoning 0 · 1 call

Tier breakdown

Each bar is the tier's mean, from 0 to 1 (for E, its mean × the excellence gate). The percentage is the tier's share of the score. Mean × share, added over all tiers, is the tests earned figure below.

L · Lint and bundles1.001.06%
K · Platform currency1.001.32%
T · Event pipeline0.8682.11%
R · Reconcile0.87951.85%
S · Storage0.99851.06%
B · Resolvers and permissions0.7871.58%
U · UI function0.47892.11%
V · Visual1.001.06%
A · Rovo0.6251.06%
E · Excellence0.82661.8%
R1 · Migration1.0012%
R2 · Dosing0.7513%
R3 · Time limits1.008%
R4 · Changing world0.901110%
R5 · Admin panel1.0010%
R6 · CI web trigger1.008%
R7 · Custom field0.82165%
R8 · Forge LLM1.004%
R9 · Boot budget1.005%
R10 · Native JQL0.490710%

Scoring detail

How this score was built

Per-test scorer output, as posted by the app. Expand a section to inspect its tests.

Correctness combines the lowest score for each test across the scoring fixtures; economy measurements use their mean. The combined result can therefore be lower than every individual fixture’s final score. Critical defects reduce the weighted result, followed by the reliability multiplier, before any score cap applies.

The rules behind these numbers: Forge 2.0 requirements, weights and score caps.

The score, step by step

0.8713 tests earned × 1.0000 critical defects × 0.4691 reliability = 0.4087 final score

tests earned = 0.15 × 0.8234 v1 + 0.7478 v2 (out of 0.85)

v1 = 0.88 × 0.8230 tiers L–A + 0.12 × 0.8266 gate × 1.0000 excellence

Reliability: every tier in the breakdown above, except Excellence, is a group of tests. Each group of tests multiplies the score by 1 − 0.10 × the shortfall of its worst test: by 0.90 when that test fails completely, by 0.95 when it scores 0.5.

  • T Event pipeline: worst test t_reestimate_followed, score 0.1667 × 0.9167

    Already counted in this group: t_out_of_order, t_multi_sprint_parse

  • R Reconcile: worst test r_heal_dropped, score 0.25 × 0.9250

    Already counted in this group: r_pagination

  • S Storage: worst test s_entity_index_used, score 0.9941 × 0.9994
  • B Resolvers and permissions: worst test b_comment_adf_as_user, score 0 × 0.9000

    Already counted in this group: b_hidden_count

  • U UI function: worst test u_not_started, score 0 × 0.9000

    Already counted in this group: u_widget_numbers, u_widget_chart, u_ledger_table, u_ledger_sort, u_modal_close, u_widget_live, u_llm_explain

  • A Rovo: worst test a_action_result, score 0 × 0.9000

    Already counted in this group: a_action_permissions

  • R2 Dosing: worst test r2_person_never_quota_refused, score 0 × 0.9000
  • R4 Changing world: worst test r4_field_switch, score 0.5962 × 0.9596

    Already counted in this group: r4_move_new_board

  • R7 Custom field: worst test r7_values_fresh, score 0.7681 × 0.9768

    Already counted in this group: r7_writes_accepted

  • R10 Native JQL: worst test r10_precomputation_refresh, score 0 × 0.9000

    Already counted in this group: r10_initial_membership, r10_shared_permissions, r10_empty_and_closed, r10_invalid_arguments

v1 is this run's score on the tests carried over from Forge 1.0: 15% of the total. v2 is the requirements R1–R10, each at its share of the score: the other 85%. Critical defects multiply the whole score, and so does each group of tests, by 1 − 0.10 × the shortfall of its worst test.

Excellence gate 0.8266: the average of the 18 conditions below, each counted at the value shown. Green is met in full; red is not, and lowers the gate. The 3 excellence tests are paid in proportion to the gate; together they are 1.8% of the total.

t_event_rows 1.00r_heal_dropped 0.25u_widget_numbers 0.375u_ledger_table 0.2535b_comment_adf_as_user 0.00k_consumer_shape 1.00k_current_apis 1.00k_dashboard_widget 1.00k_entity_declared 1.00k_llm_model_current 1.00k_manifest_semantics 1.00k_rovo_mcp 1.00k_rovo_skill 1.00k_runtime_risks 1.00k_widget_edit_bridge 1.00v_console_clean 1.00v_csp_clean 1.00l_lint_warnings 1.00

Earned score 0.4087 · No cap applies · Final score 0.4087

Under a cap the final score is min(earned, cap − 0.05 × (1 − earned)): always below the cap, and closer to it the more the run earned. With no cap it is the earned score. Avoiding a cap adds no points.

The dashboard widget's numbers and chart, its board choice saved through the host, live updates through Forge Realtime, the sprint action's ledger table, sorting, issue links, comment flow, the Forge LLM explanation, close and not-started states.

u widget loadsdiagnostic, no weight

1.00

6/8 configured widget views render a sprint with its numbers

u widget numbers

0.375

worst of 3 scoring sites (seed 3aaa6d6b73d37669): widget numbers: 3.00/8 views exact (sprints by startDate, four §1 numbers each)

u widget chart

0.25

worst of 3 scoring sites (seed 3aaa6d6b73d37669): 2/8 charts on one linear scale within 1px; ['worst bar off its linear scale by 1.04px', 'worst bar off its linear scale by 1.05px']

u widget edit config

1.00

edit/config 10/10:

u ledger table

0.2535

worst of 3 scoring sites (seed d031c3ee22136508): ledger tables: 1.52/6 exact (rows, cells, default order)

u ledger sort

0.50

9/18 at-toggle states ordered with aria-sort (6 after another sort was active)

u issue router

1.00

6/6 issue keys open /browse/<KEY> through the Forge router

u comment flow

1.00

32/32 comment-flow steps right (select, one comment and a success flag per click and double click, forbidden error flag, modal works)

u modal close

0.3333

worst of 3 scoring sites (seed d031c3ee22136508): 2/6 close clicks reached bridge close

u not started

0.00

worst of 3 scoring sites (seed d031c3ee22136508): future sprint context: not-started not shown alone

u widget live

0.00

moved sprints showing the new numbers live 0/1; 0 idle invoke(s)

u llm explain

0.5556

worst of 3 scoring sites (seed 3c04f61f28a4a511): explain 5/9 over 5 scripted answers: 1:digits:one_llm_call_answered_digits, 2:refusal:one_llm_call_answered_refusal, 3:malformed:one_llm_call_answered_malformed, 4:error:

Graded browser recording

Loading duration…

Watch the full graded browser recording at its original speed: the dashboard widget, the sprint action, the not-started sprint and the Forge LLM cases, as the scorer drove them on the run's own seed, the first of the three scoring sites. The test results and screenshots on this page provide the wider evidence. Pause or scrub to inspect a view.

The UI Kit admin panel is not in the recording: it is graded from its component tree.

Ready to play0:00 / —:—
Open recording file

Drag the timeline to seek. Keyboard: arrow keys move 5 seconds; Home and End jump to the start and end.

Full graded browser recording: dashboard widget (before configuration, edit and Save, light and dark at 380 px, a second instance at 1180 px), sprint action (light and dark, sort, issue link, select, comment post, double click, forbidden post, explain, close), the not-started sprint and the Forge LLM cases, where the app has each. The UI Kit admin panel is not in the recording.

Screenshots

Pictures from one of the three scoring sites, the one on the run's own seed. The dashboard widget and the sprint action are captured from the built app in a browser during grading. The admin panel picture is drawn by the benchmark from the app's component tree, not by Jira, and nothing is graded from that picture. A test keeps the score of its worst site (the excellence measurements keep their mean), so a test can fail on a site these pictures do not show. A detail that opens "worst of …" names the seed of the site the test scored worst on, which can be the pictured one.

1Dashboard widget · light
2Dashboard widget · dark
3Sprint action · light
4Sprint action · dark
5Admin panel (UI Kit) · light: drawn by the benchmark from the app's component tree, not by Jira.

Run details

Model
qwen/qwen4-max
Engine events
0
Repair rounds
0
Started
Nov 16, 2026, 12:07 PM
Finished
Nov 16, 2026, 01:31 PM
Call budget
With wrap-up nudge, version 2: as the call budget ran low, the agent harness told the model to start no new work and finish

How the benchmark is scored