Swarm baseline

Gemini 3.8 Flash — 0.6990 on Gauntlet 7.1 pilot

gemini-3.8-flash · 1 node · scorer sb-7.1-rc

Gemini 3.8 Flash · SB7.1 pilotSep 20, 2026
Overall
0.6990
Excellent
no
Benchmark
Gauntlet 7.1 pilot
Model build
9m 14s
Nodes
1

Tier breakdown

A · structure & runtime1.00
B · behaviour1.00
C · vendor contract1.00
D · finesse0.83

Scoring detail

How this score was built

Per-check scorer output, as posted by the app. Expand a section to inspect its checks.

Earned credit before admission

(0.88 × 0.9263 core + 0.12 × 0.94 gate × 0.75 excellence) × 0.8857 critical = 0.7968

Critical defects compound a multiplier on the whole score (pre-severity 0.8996):

j_workflow_journey 0.71 → ×0.89

The core (88% of the total) is the weighted mean of the ten measured tiers. The last 12% is the excellence slice: it unlocks in proportion to the perfection conditions below (14 of 16 met here), then pays out at the excellence tier's own measured mean. Core 0.8151 + excellence 0.0845.

j_first_use 1.00j_workflow_journey 0.71j_error_state 0.30j_empty_state 1.00console_clean 0.00v_responsive_375 1.00v_dates_readable 1.00t_scene_binding 1.00x_conservation_residual 1.00r_no_row_loss 1.00p_drag_frames 1.00p_idle_flatness 1.00p_stream_apply 1.00p_under_stream 1.00p_api_latency 1.00p_sync_wall 1.00

Earned score 0.797 · Admission ceiling 0.699 · Final score 0.699

Final score is the lower of earned credit and the admission ceiling. Passing admission adds no points.

Visible: passedMatching: not metGood: not metVisual excellence: not metBackend excellence: not met

Required payment-tower structure or currency mapping is incomplete

Payment context, readability or interaction quality is incomplete

Scores above 0.9 require both verified event animation and backend recovery excellence

Run notes and corrections

SB7.1 pilot, not a stable release or an unrestricted model ranking. Actual model: gemini-3.8-flash. App-launched single-agent session cloud-b3c625d6-1c5e-4c3f-8b5b-e2b154d7ae59, 20 September 2026. Candidate source was preserved unchanged. Final external CLI scorer commit 3f6f7ac1b; fixture seed 5cd00e961d80464f. Canonical report SHA-256 cbb5c823f319011f84d13fb537429ca61d2a4769ef797fac11a18464e400ed14. Model build: 554 seconds (9m14s), from the isolated session timestamps. Final grading pass: 226.297 seconds (3m46s). Wall time shown on this card is model build time, excluding operator debugging and rejected grading attempts. Earlier scorer failures were corrected and the unchanged artifact was rescored. Visible payment towers, geometry and currency mapping passed. The 0.699 ceiling is caused by scene-truth/rest-pixel failures, independently traced to fixed 1/60-second camera coast steps instead of actual elapsed frame time. The generic admission reason below says structure/mapping; this is not a finding that tower parts were absent. Raw earned credit: 0.7968. Actual defects: a stale event visibly regresses v4 to v3, and atomicity/conservation probes observed 26 half-applied backend states in 815 samples. Presentation earned partial credit. Remaining measurement caveats: annotation exit checks wrongly demand the hidden attribute when empty content can also be invisible; RAF-to-draw trajectory timing needs further validation. Those M checks have zero weight in raw credit; stale-event regression independently prevents animation excellence. The candidate sandbox blocked its Chrome self-test. External grading succeeded with no missing infrastructure or unavailable probes, but this limited developer-tools profile must be disclosed. Shared-host load also limits performance comparisons. Screenshots and video are from the actual graded Gemini app, not the reference. Recorded usage: 9,596,417 input tokens including 5,702,519 cached input; 75,873 candidate output tokens; 9,731,669 total tokens. Provider output accounting omits thinking. Reconstructed cost is approximately $3.63–$3.86 at the standard introductory rates checked on 20 September 2026; not a verified invoice, not zero cost, and not a proven saving against SB7. Pricing source: https://ai.google.dev/gemini-api/docs/pricing .

Graded browser recording

Loading duration…

Watch an excerpt of the generated application during browser grading. The check results and screenshots on this page provide the wider evidence. Pause or scrub to inspect a view.

Each tower represents a payment: height encodes its amount, cap color its status, and the collar its currency. The selected detailed tower is expected to animate its collar after a committed backend update or replay. These are the task requirements; the check results show what this app actually achieved.

Ready to play0:00 / —:—
Open recording file

Drag the timeline to seek. Keyboard: arrow keys move 5 seconds; Home and End jump to the start and end.

Actual Gemini pilot: payment inspection, live update and replay checks. Includes observed failures.

Screenshots

Captured from the built application during browser grading. Captions identify the recorded views and checks; these images do not represent repair rounds.

1Gemini-built payments console after synchronization
2Gemini-built data-backed payment tower field
3Payment inspection with currency and status context
4Recorded committed-update inspection
5Gemini-built console at mobile width

Run details

Model
gemini-3.8-flash
goose build
3f6f7ac1b

How the benchmark is scored