Cloud baseline

Hy4 flash, single model via OpenRouter

tencent/hy4-flash · scorer forge-2.0

tencent/hy4-flash · single agentNov 17, 2026
Overall
0.5103
Benchmark
Forge 2.0
Model build
27m 1s

Recorded inference cost

$0.11

From the saved OpenRouter generation billing summary (244 recorded requests). Whether auxiliary model calls were included was not retained. Scoring reuses the saved build; this amount describes the original inference spend.

Token usage

Tokens: 43.7M in · 103k out · 42.8M cached (exact, OpenRouter records)

43,842,648 tokens over 244 model calls: every call of the run, the model's own and the harness's helper calls, counted from OpenRouter records. Reasoning tokens as the provider reports them: 42,308 (not added to the total).

  • tencent/hy4-flash

    In 43.7M · Out 103k · Cached 42.8M · Reasoning 42.3k · 243 calls

  • google/gemini-2.5-flash

    In 1.4k · Out 5 · Cached 1k · Reasoning 0 · 1 call

Tier breakdown

Each bar is the tier's mean, from 0 to 1 (for E, its mean × the excellence gate). The percentage is the tier's share of the score. Mean × share, added over all tiers, is the tests earned figure below.

L · Lint and bundles1.001.06%
K · Platform currency0.9751.32%
T · Event pipeline0.97922.11%
R · Reconcile0.99811.85%
S · Storage1.001.06%
B · Resolvers and permissions0.86111.58%
U · UI function0.70312.11%
V · Visual1.001.06%
A · Rovo0.501.06%
E · Excellence0.94941.8%
R1 · Migration1.0012%
R2 · Dosing0.9013%
R3 · Time limits1.008%
R4 · Changing world0.763610%
R5 · Admin panel0.966710%
R6 · CI web trigger1.008%
R7 · Custom field0.91985%
R8 · Forge LLM1.004%
R9 · Boot budget1.005%
R10 · Native JQL0.62510%

Scoring detail

How this score was built

Per-test scorer output, as posted by the app. Expand a section to inspect its tests.

Correctness combines the lowest score for each test across the scoring fixtures; economy measurements use their mean. The combined result can therefore be lower than every individual fixture’s final score. Critical defects reduce the weighted result, followed by the reliability multiplier, before any score cap applies.

The rules behind these numbers: Forge 2.0 requirements, weights and score caps.

The score, step by step

0.9031 tests earned × 1.0000 critical defects × 0.5651 reliability = 0.5103 final score

tests earned = 0.15 × 0.8969 v1 + 0.7685 v2 (out of 0.85)

v1 = 0.88 × 0.8897 tiers L–A + 0.12 × 0.9494 gate × 1.0000 excellence

Reliability: every tier in the breakdown above, except Excellence, is a group of tests. Each group of tests multiplies the score by 1 − 0.10 × the shortfall of its worst test: by 0.90 when that test fails completely, by 0.95 when it scores 0.5.

  • K Platform currency: worst test k_widget_edit_bridge, score 0.75 × 0.9750
  • T Event pipeline: worst test t_reestimate_followed, score 0.8333 × 0.9833
  • R Reconcile: worst test r_pagination, score 0.9851 × 0.9985
  • B Resolvers and permissions: worst test b_hidden_count, score 0.3333 × 0.9333

    Already counted in this group: b_comment_adf_as_user

  • U UI function: worst test u_widget_live, score 0 × 0.9000

    Already counted in this group: u_widget_numbers, u_widget_chart, u_widget_edit_config, u_ledger_table, u_ledger_sort, u_comment_flow, u_llm_explain

  • A Rovo: worst test a_action_result, score 0 × 0.9000

    Already counted in this group: a_action_permissions

  • R2 Dosing: worst test r2_background_share, score 0.6 × 0.9600
  • R4 Changing world: worst test r4_permission_revoked, score 0 × 0.9000

    Already counted in this group: r4_field_switch

  • R5 Admin panel: worst test r5_panel_labels, score 0.9 × 0.9900
  • R7 Custom field: worst test r7_values_fresh, score 0.8682 × 0.9868

    Already counted in this group: r7_writes_accepted

  • R10 Native JQL: worst test r10_empty_and_closed, score 0.25 × 0.9250

    Already counted in this group: r10_initial_membership, r10_precomputation_refresh, r10_shared_permissions

v1 is this run's score on the tests carried over from Forge 1.0: 15% of the total. v2 is the requirements R1–R10, each at its share of the score: the other 85%. Critical defects multiply the whole score, and so does each group of tests, by 1 − 0.10 × the shortfall of its worst test.

Excellence gate 0.9494: the average of the 18 conditions below, each counted at the value shown. Green is met in full; red is not, and lowers the gate. The 3 excellence tests are paid in proportion to the gate; together they are 1.8% of the total.

t_event_rows 1.00r_heal_dropped 1.00u_widget_numbers 0.625u_ledger_table 0.8802b_comment_adf_as_user 0.8333k_consumer_shape 1.00k_current_apis 1.00k_dashboard_widget 1.00k_entity_declared 1.00k_llm_model_current 1.00k_manifest_semantics 1.00k_rovo_mcp 1.00k_rovo_skill 1.00k_runtime_risks 1.00k_widget_edit_bridge 0.75v_console_clean 1.00v_csp_clean 1.00l_lint_warnings 1.00

Earned score 0.5103 · No cap applies · Final score 0.5103

Under a cap the final score is min(earned, cap − 0.05 × (1 − earned)): always below the cap, and closer to it the more the run earned. With no cap it is the earned score. Avoiding a cap adds no points.

The get-sprint-scope action returns exact numbers and the visible changes for the invoking person, returns errors instead of throwing, and the skill's SKILL.md tells the agent how to use it.

a action result

0.00

0/5 active sprints answered exactly

a action errors

1.00

2/2 bad sprintId inputs return {error} without throwing

a action permissions

0.00

0/12 per-person answers list exactly what that person may browse

a skill instructions

1.00

SKILL.md body 5/5: result fields named ['sprintName', 'committed', 'added', 'removed', 'creepPercent', 'hiddenChanges', 'changes']

Graded browser recording

Loading duration…

Watch the full graded browser recording at its original speed: the dashboard widget, the sprint action, the not-started sprint and the Forge LLM cases, as the scorer drove them on the run's own seed, the first of the three scoring sites. The test results and screenshots on this page provide the wider evidence. Pause or scrub to inspect a view.

The UI Kit admin panel is not in the recording: it is graded from its component tree.

Ready to play0:00 / —:—
Open recording file

Drag the timeline to seek. Keyboard: arrow keys move 5 seconds; Home and End jump to the start and end.

Full graded browser recording: dashboard widget (before configuration, edit and Save, light and dark at 380 px, a second instance at 1180 px), sprint action (light and dark, sort, issue link, select, comment post, double click, forbidden post, explain, close), the not-started sprint and the Forge LLM cases, where the app has each. The UI Kit admin panel is not in the recording.

Screenshots

Pictures from one of the three scoring sites, the one on the run's own seed. The dashboard widget and the sprint action are captured from the built app in a browser during grading. The admin panel picture is drawn by the benchmark from the app's component tree, not by Jira, and nothing is graded from that picture. A test keeps the score of its worst site (the excellence measurements keep their mean), so a test can fail on a site these pictures do not show. A detail that opens "worst of …" names the seed of the site the test scored worst on, which can be the pictured one.

1Dashboard widget · light
2Dashboard widget · dark
3Sprint action · light
4Sprint action · dark
5Admin panel (UI Kit) · light: drawn by the benchmark from the app's component tree, not by Jira.

Run details

Model
tencent/hy4-flash
Engine events
0
Repair rounds
0
Started
Nov 17, 2026, 05:28 PM
Finished
Nov 17, 2026, 06:40 PM
Call budget
With wrap-up nudge, version 2: as the call budget ran low, the agent harness told the model to start no new work and finish

How the benchmark is scored