Cloud baseline
upstage/solar-mini4, single model via OpenRouter — 0.0083 on Gauntlet 7.2
upstage/solar-mini4 · scorer sb-7.2
- Overall
- 0.0083
- Excellent
- no
- Benchmark
- Gauntlet 7.2
- Model build
- 24m 3s
- Prompt tok
- 11.0M
- Gen tok
- 87.6k
Tier breakdown
Scoring detail
How this score was built
Per-check scorer output, as posted by the app. Expand a section to inspect its checks.
Earned credit before admission
(0.88 × 0.0158 core + 0.12 × 0.00 gate × 0.00 excellence) × 0.6000 critical = 0.0083
Critical defects compound a multiplier on the whole score (pre-severity 0.0139):
sync_completeness: no additional penalty (root:server_runs); data loss — silently missing payments.
b_money_rendered: no additional penalty (vacuous:sync_completeness); wrong money — wrong exponent/digits or a cross-currency sum.
b_buckets_dst: no additional penalty (vacuous:sync_completeness); wrong money — mis-bucketed days.
j_loads_data: no additional penalty (root:server_runs); dead primary flow — no data visible.
j_workflow_journey: no additional penalty (root:server_runs); dead primary flow — approval cannot complete through the UI.
x_conservation_residual: no additional penalty (vacuous:sync_completeness); wrong money — unexplained minor units created/destroyed after dupe/loss attribution.
x_no_lost_write: no additional penalty (vacuous:sync_completeness); wrong money — an acknowledged mutation absent from final state.
r_no_row_loss: no additional penalty (vacuous:sync_completeness); data loss — a committed row missing after any seeded kill.
r_no_dupe_effect: no additional penalty (vacuous:sync_completeness); wrong money — a ledger effect applied twice.
r_cache_truth: no additional penalty (vacuous:sync_completeness); data loss — 304-vs-cache mismatch served as fresh.
r_workflow_durability: no additional penalty (vacuous:server_runs); data loss — submitted/approved state reverting after SIGKILL.
The core (88% of the total) is the weighted mean of the thirteen measured tiers. The last 12% is the excellence slice: it unlocks in proportion to the perfection conditions below (0 of 16 met here), then pays out at the excellence tier's own measured mean. Core 0.0139 + excellence 0.0000.
Earned score 0.008 · Admission ceiling 0.599 · Final score 0.008
Final score is the lower of earned credit and the admission ceiling. Passing admission adds no points.
Visible data-backed 3D: s_visible_surface (maximum 0.599)
Tower structure, currency mapping and scene truth: s_visible_surface, s_tower_geometry, s_currency_collar, t_layout_basis, t_scene_binding, t_height_pixels, t_vs7dbg_truth (maximum 0.699)
3D interaction and overview legibility: t_draw_budget, t_pick_buffer, t_pick_real_pass, t_click_semantics, t_camera_math, t_coast_identity, t_coast_reality, t_labels_culling, t_brush_link, t_stream_diff, q_overview_legibility, q_inspector_framing (maximum 0.799)
Event animation and backend recovery: m_committed_event_replay, x_l1_no_invented_states, x_l2_per_key_order, x_l3_monotonic_reads, x_l4_convergence, x_l5_group_atomicity, x_m1_amount_immutability, x_m2_pair_conservation, x_m3_terminal_conservation, x_m4_no_cross_currency, x_conservation_residual, x_no_lost_write, x_ooo_dup_forged, r_b3_sigkill_resync, r_b4_vendor_down_boot, r_b6_outbox_atomic, r_b7_partition, r_notifier_exactly_once, r_notification_multiset, r_no_row_loss, r_no_dupe_effect, r_cache_truth, r_workflow_durability (maximum 0.899)
the two services boot and bind, the spec-named files exist, the 150 KB asset budget holds, each service owns only its own database
a package layout
0.88The submitted tree contains every deliverable the spec names by path — app/__main__.py, DECISIONS.md, and the four web files (index.html, styles.css, app.js, viz.js) — plus a bootable module for each of the two services.
5/6 named files, 2/2 bootable service modules
server runs
0.15Both services boot as real processes and report healthy over HTTP within the spec's 10-second budget — the tool runs at all.
process survives 5s without binding
a combined entrypoint
0.00The documented single-command form — python -m app — really boots BOTH services, not just the two individual service commands the harness normally uses.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
serves page
0.00The backend itself serves the frontend: GET / returns the page and styles.css, app.js, viz.js come back with correct content types, as a real browser would need.
GET / -> None
a asset budget
1.00The whole frontend fits the 150 KB source budget and ships zero external code — the budget exists so a vendored 3D library cannot, and the page must work fully offline.
47 KB of 150 KB (4/4 files), 0 external ref(s)
a db ownership
0.50Each service owns exactly its own SQLite file under --db-dir — ledgerd's ledger.db and notifierd's notifier.db — the one-file-per-service contract.
ledger.db=True notifier.db=False
what the API serves on the wire: sync completeness (12,288 rows), money and Berlin-DST bucketing, the append-only event ledger, error envelopes
sync completeness
0.00After the self-driven full sync the app's local store holds every one of the vendor's 12,288 fixture payments — an incomplete sync silently loses money rows.
Eventual evidenced total 0; initial API snapshot total None; initial expected total None. Totals reflect their respective measurement phases.
b total field
0.00GET /api/payments reports the correct collection total — a wrong total breaks every caller's paging math.
no full state to grade the total against
b row shape
0.00A payment row on the wire carries exactly the 10 documented keys (id, amount_minor, currency, created_at, settled_at, status, version, note, counterparty_name, country) — the vendor's nested counterparty object flattened into the last two.
no rows
b chronological order
0.00The default payments page is ordered by created_at as a parsed instant, not as a string — mixed UTC offsets make string ordering wrong.
too few rows
b summary shape
0.00GET /api/summary carries the documented per-currency money blocks: by_currency and the reversals block, both sorted ascending by currency code — the summary is the money surface.
no summary
b money rendered
0.00Rendered money is arithmetically correct per currency — right digits and right decimal exponent, with the zero-decimal JPY and three-decimal KWD traps weighted separately — and no cross-currency sum exists anywhere in the summary.
vacuous — nothing to lose because `sync_completeness` already failed
b buckets dst
0.00GET /api/buckets — the 3D field's aggregate — buckets payments into Europe/Berlin calendar days per status, correct across the seeded DST transition (the data-correctness trap).
vacuous — nothing to lose because `sync_completeness` already failed
b viz records
0.00GET /api/viz/records — the 3D field's sanctioned full fetch — serves the whole collection columnar with the server-computed Berlin day, so the frontend never recomputes days in UTC.
no /api/viz/records
b events log
0.00The append-only event ledger is contiguous from seq 1 with the frozen type and source vocabularies — a seq gap is evidence of a lost write and is graded as one.
no events returned (status None)
b error envelope
0.00Error responses carry the documented structured envelope — error.code and error.message plus field_errors[] with dot-and-[index] paths where validation detail is expected.
no error responses observed
b json shapes
0.00Every sampled API response — success and error paths alike — is parseable JSON; an HTML error page mid-API breaks every client that trusted the contract.
no responses sampled
how the vendor is consumed: the 192-page walk, drop resume, Retry-After, the collection-generation rule, webhook and idempotency-key discipline
c paged walk
0.00The first sync walks the vendor's server-fixed 64-per-page protocol to completion — all 192 pages, documented parameters only, no page fetched twice.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
c b1 drop resume
0.00A connection dropped mid-page-stream during the first walk costs a documented resume — not a full restart of committed work, and not a hole in the collection.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
c b2 retry after
0.00A seeded 500 with Retry-After during the walk costs exactly one documented retry after the advertised wait — never a fresh unconditional restart of committed work.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
c b5 generation 304
0.00The collection-generation rule: a 304 whose X-Collection-Generation disagrees with the stored generation is a cache miss — drop the validator and refetch unconditionally exactly once, without looping and without serving stale data as fresh.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
c conditional resync
0.00Later syncs are cheap: every re-sync request carries a validator (If-None-Match) except the documented unconditional cases — an unconditional full re-walk on every sync is the expensive-client defect.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
c webhook discipline
0.00The app accepts the vendor's signed push traffic — every delivery acknowledged 2xx within budget — and bounces the one forged signature with 401, state untouched.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
c send idempotency
0.00Approved drafts become real vendor payments through POST /v3/payments with a stored Idempotency-Key, and the kill-interrupted send's retry reuses that key — a fresh key per retry is the seeded duplicate-payment bug.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
input validation, content types, client timeouts, behavior when the peer service is down, and the DECISIONS.md judgment corners
d content types
0.00API responses declare a JSON content type and the SSE stream declares text/event-stream — a wrong content type breaks strict clients and kills EventSource.
api json 0.00, /api/stream ctype ''
d validation
0.00Invalid input and wrong-role requests are rejected with the documented status codes — bad limit/offset/sort/status 400, unknown paths 404, missing tokens 401, wrong roles 403, self-approval 403 approval_forbidden.
matrix not exercised
d client timeouts
0.40The app is provably resilient to an unresponsive vendor: it boots, binds and serves local data while the vendor refuses connections — one hung vendor call must never hang the tool.
vendor-down boot not proven; timeout= present in source
d peer absence
0.00Neither service crashes on the other's absence: with notifierd down the proxy answers 502 with the documented envelope code and ledgerd keeps running; with ledgerd killed notifierd keeps running.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
d decisions doc
0.00The three deliberately unstated corners — D1 brush survival on streamed mutation, D2 rejected-draft terminality, D3 pre-first-sync table state — are decided, documented under the frozen headings in DECISIONS.md, and consistent with observed behavior.
DECISIONS.md absent or carries none of the frozen ## D1/D2/D3 headings
real user flows in a real browser: first use, the sync click, the maker/checker approval journey, notifications, error and empty states
j loads data
0.00Whether the Meridian Payments Console renders any payment rows at all in a real browser, and whether the total it claims on the page matches the collection truth at probe time.
0 rows rendered, DOM claims None (want None)
j console clean
1.00Whether normal use of the console — loading it and running a sync — produces JavaScript console errors or uncaught page errors.
0 console error(s) across load+sync
j first use
0.00The first-visit experience as one property: real data appears quickly, the on-page total agrees with the collection truth, and the console stays clean.
ttfd=Nonems reconcile=0.0 consoleErrs=0
j sync journey
0.00The headline interactive flow: a user clicks Sync now and the app visibly starts, indicates progress, finishes, and refreshes the view.
sync button never found
j workflow journey
0.00The maker/checker approval workflow driven end to end through the UI alone: token in, draft created, submitted, approved, listed, notified, and the vendor-created payment landing in the payments table.
required app surface absent: workflow UI (role token / draft form / approve)
j workflow reject
0.00The checker's other half of the workflow: rejecting a submitted draft completes through the UI, the rejected state shows in the draft list, and the rejection notification appears in the feed.
rejection never completed
j notifications feed
0.25The notifications feed visibly degrades while notifierd is down and heals itself — without a reload — once it returns.
partition state=None, heal state=None, live in Nones
j error state
0.00What a user sees when the backend is unreachable: a visible, actionable error state instead of a blank or silently broken page.
no error state
j empty state
0.00What a fresh install looks like before the first sync completes: an honest empty-or-progress state rather than phantom data or a blank page.
blank page on empty db
what the page visibly shows: human dates, exponent-correct money, status badges at the frozen hexes, 375 px layout, deliberate styling
v dates readable
0.00Whether the Date column shows humans a readable date instead of a raw machine timestamp.
no date cells rendered
v money presentation
0.00Whether rendered amounts identify their currency — the presentation half of money; the exponent-and-digits truth is graded separately by the critical b_money_rendered.
no amount cells rendered
v status badges
0.00Whether the four payment statuses are visually distinguishable at a glance and painted in the spec's frozen palette — the same four hexes the 3D field uses.
no status cells rendered
v responsive 375
0.00Whether the page survives a phone-width viewport: no horizontal scrolling and real content still rendered at 375 px.
375px horizontal scroll: None
v styling
0.20Whether the page is deliberately styled rather than browser-default: a real stylesheet, layered surface colors, a chosen font, and a branded header.
stylesheet=False, None backgrounds, font '', header=True
v presentation text
0.00Required SB7.2 presentation observation was not reached
the spec's budgets, measured: frames under drag, idle flatness (demand rendering), stream apply, API latency under load, sync wall clock
p drag frames
0.00The 3D field stays interactive under input: real frames rendered during the scripted 40-move budget drag at the full 12,288-instance count.
frames not measurable over the scripted drag
p idle flatness
0.00Demand rendering, the frozen spec rule: at rest — no input, no coast, no pending stream batch — the scene draws nothing; a continuous rAF render loop fails by design.
no idle windows sampled
p stream apply
0.00A live SSE batch becomes visible fast: the median time from receipt to applied — store, digest and pixels — against the spec's 250 ms budget.
no stream batch applied
p under stream
0.00The API stays fast AND correct while the SSE stream burst is landing — read p95 under proven concurrent load.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
p api latency
0.00The read endpoints answer within their spec budgets when idle: the worst p95 across the API latency battery.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
p sync wall
0.00Wall-clock time for the self-driven first sync — the full 192-page walk, seeded faults and documented waits included — against the spec's 120 s budget.
sync #1 never completed inside the harness budget
the instanced WebGL field: real context, scene math, GPU pick buffer, camera and coast physics, collision-culled labels, brush, streaming diffs
t context real
0.00The #viz3d panel is a real, drawing WebGL surface with the pinned context attributes and a correctly sized backing store — not a styled div, an image, or an unused canvas.
ctx=0 type=None nonBg=None/None distinct=None backingOk=None
t layout basis
0.00The locked layout basis vs7dbg.layout() reports — d0 (first day), D0 = 96 (the span), R0 (max in-day count at load) — matches the fixture truth and never moves when a streamed create arrives.
layout got={'d0': None, 'D0': None, 'R0': None} want={'d0': '2026-08-19', 'D0': 96, 'R0': 178} unmoved_after_stream=None
t scene binding
0.00The rendered scene actually encodes the payment data: vs7dbg.sceneDigest()'s seven statistical moments over all 12,288 instanced columns match an independent recomputation from the fixture.
no data to bind
t height pixels
0.00Column heights are true in rendered pixels — including the JPY (exponent 0) and KWD (exponent 3) instances whose heights expose a forgotten currency exponent.
no data to measure
t draw budget
0.00The field renders 12,288 instances inside the draw budget — at most 8 default-framebuffer draw calls per rendered frame — forcing instanced draws or a merged buffer instead of per-column draws.
wrapper counted no draws over the budget window
t pick buffer
0.00Picking is GPU truth: the offscreen pick buffer answers occlusion exactly as the depth buffer says, on click points constructed to kill CPU raycasts and last-drawn-wins shortcuts.
no instances to pick
t pick real pass
0.00The pick buffer is real GPU work with the documented cost profile: a fresh offscreen pass after each scene invalidation, bounded at 4 offscreen draws, and never flashing ID colors on the visible canvas.
no pick-refresh counter windows
t click semantics
0.00Canvas clicks follow the frozen §3.3 semantics: click an instance to toggle it into the brush, click it again to toggle it out, click background to clear the brush.
click semantics never exercised
t camera math
0.00The orbit camera implements the documented math exactly: defaults yaw 30 / pitch 40 / distance 260, the 0.30 deg-per-px drag law, the exponential wheel law without page scroll, both clamps, double-click reset, and a projection that matches the printed formula.
camera math never exercised
t coast identity
0.00Post-release inertia obeys the closed-form τ = 0.4 s decay law: at any coasting instant the remaining travel equals v(t)·τ, a slow release starts no coast, and the coast settles inside the printed budget.
no coast evidence at all
t coast reality
0.00The coast is real motion on the canvas, not a camera() narrative: after a fast flick the scene provably keeps moving past release.
no flick-coast evidence
t labels culling
0.00The 12 highest-amount records get screen-space labels that are collision-culled deterministically THROUGH the app's own pick buffer — exact set, exact geometry, zero overlap, never floating over an occluded instance.
no candidates to label
t brush link
0.00One brush set links the table and the 3D field in both directions: row clicks and instance clicks toggle the same set, non-members dim to the exact 0.30 pixel rule, the count readout tracks, and background click restores full color.
no rows to brush
t stream diff
0.00SSE batches apply as true diffs: only the changed instances upload, no buffer realloc, the digest moves by exactly the change, and the changed instance's pixels show it.
no SSE batch observed applying
t vs7dbg truth
0.00The mandated window.vs7dbg instrumentation tells the truth: camera() agrees with the pixels, sceneDigest() with the recomputed data, frames() with the wrapper's counted draws, and pick() with pickPixel() with the analytic answer.
required app surface absent: window.vs7dbg
the live consistency contract: no invented states, per-key version order, monotonic reads, convergence, money conservation, dup/forgery handling
x l1 no invented states
0.00Every (payment, version) the app ever applied or served exists in the vendor's committed history — an invented state means the app fabricated data.
nothing applied to check
x l2 per key order
0.00Applied versions per payment strictly increase in event-log order — duplicate and stale webhook outcomes belong in the counters, never as events.
no event log to order-check
x l3 monotonic reads
0.00The version served for a payment never decreases from one read to the next — a sync page landing after a webhook applied v+1 must not regress the row.
no read stream against a live sync
x l4 convergence
0.00At quiescence every payment's version and status equals the vendor's final committed state, and the row count equals the vendor's — the mid-walk create present exactly once.
never converged because sync never completed
x l5 group atomicity
0.00No read ever observes half a transaction group: the refunded payment and its reversal become visible together or not at all.
the group never applied
x m1 amount immutability
0.00No served row ever shows an amount_minor different from the vendor-committed amount — v3 never mutates amounts, only status/note/version.
no amounts served
x m2 pair conservation
0.00At every observed instant, per currency, the summary's reversal totals equal the refunded rows' amounts — both halves of each refund visible, or neither.
no summaries to conserve
x m3 terminal conservation
0.00Terminal per-currency counts and totals — reversals included — equal vendor ground truth: fixture plus scripted mutations plus every payment the app created.
no terminal state
x m4 no cross currency
0.00No field anywhere in the summary carries a cross-currency money sum — minor units are not a common denomination and summing them is wrong money.
no summary to scan
x conservation residual
0.00CRITICAL — after every duplicate and loss is attributed, no minor units remain created or destroyed: the money conservation residual is zero in every currency.
vacuous — nothing to lose because `sync_completeness` already failed
x no lost write
0.00CRITICAL — every mutation the app acknowledged with a 2xx is present in the final state; an acked-then-vanished write is silent data loss.
vacuous — nothing to lose because `sync_completeness` already failed
x ooo dup forged
0.00The three webhook trust traps land correctly: the out-of-order pair keeps v+2 (the late v+1 never overwrites), the forged-signature event changes nothing, and no duplicate delivery applies twice.
the deliveries never applied
the seeded fault schedule: SIGKILL mid-sync, vendor-down boot, outbox atomicity, partition catch-up, exactly-once effects, workflow durability
r b3 sigkill resync
0.00ledgerd SIGKILLed mid-sync restarts, converges, and duplicates nothing — a kill mid-walk costs a clean cursor restart, never dupes or holes.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
r b4 vendor down boot
0.00The app boots with the vendor down: binds within 10 seconds anyway, serves local data, does not crash, and completes the first sync unprompted once the vendor returns.
{'armed': True, 'refusal_secs': 4, 'bound_in_10s': False, 'served_while_down': False, 'no_crash': False}
r b6 outbox atomic
0.00A SIGKILL between an outbox commit and its delivery loses nothing and doubles nothing — the exact window where commit-then-POST and POST-then-commit both fail.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
r b7 partition
0.00A notifier partition degrades visibly and heals in order: writes never block, /api/outbox/status reports down with growing pending, the feed shows degraded, and catch-up is in seq order with the UI live again within 5 seconds of heal.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
r notifier exactly once
0.00The notifier's durable processed set proves exactly-once: every outbox-crossing event processed, no seq twice, surviving kills.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
r notification multiset
0.00Selective materialization is exact: draft.submitted, draft.approved, draft.rejected and reversal.created each produce exactly one notification row; payment.sent produces none.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
r no row loss
0.00CRITICAL — no committed row goes missing after any seeded kill: the row count after every SIGKILL+restart is at least the count before it.
vacuous — nothing to lose because `sync_completeness` already failed
r no dupe effect
0.00CRITICAL — no ledger effect applies twice: one vendor payment per approved draft, no duplicate notification rows, no (type, payment, version) event effect repeated.
vacuous — nothing to lose because `sync_completeness` already failed
r cache truth
0.00CRITICAL — the lying-304 trap: a 304 whose X-Collection-Generation disagrees with the stored generation is a cache miss, and serving the stale cache as fresh is graded data loss.
vacuous — nothing to lose because `sync_completeness` already failed
r workflow durability
0.00CRITICAL — submitted and approved are durable the moment their 200 is written: a SIGKILL immediately after either, including mid-send, must find the state intact after restart.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
the last 12%: drag frames, stream-apply latency, latency under load, optimistic paint, and mastery of the T+X+R mechanisms
e frames under drag
0.00Excellence-grade fluidity: frames rendered during the scripted 40-move drag at the full 12,288-instance count, with proof the frames actually drew.
frames not measurable or no draw calls
e stream apply latency
0.00Excellence-grade streaming: the median SSE batch apply time against rungs 2.5× tighter than the P-tier budget.
no stream batch applied
e under load latency
0.00The read API's p95 under the proven stream-burst load, graded again in the excellence slice.
not exercised: ledgerd never bound its port at boot, so grading stopped before this phase
e optimistic paint
0.00The workflow UI paints optimistically: the submitted state appears while the write is provably still on the wire, then really saves.
no optimistic-paint exercise in the flow emit
e mastery
0.00Excellence includes the mechanisms, not only the surface: mastery of the entire 3D contract (T), consistency-and-money invariants (X) and resilience (R) tiers together.
T+X+R mean 0.000
Payment scene geometry and data mapping
s visible surface
0.00Required SB7.2 visual observation was not reached
s tower geometry
0.00Required SB7.2 visual observation was not reached
s currency collar
0.00Required SB7.2 visual observation was not reached
Overview legibility, inspector framing and inspected payment values
q payment context
0.00Required SB7.2 visual observation was not reached
q overview legibility
0.00Required SB7.2 visual observation was not reached
q inspector framing
0.00Required SB7.2 inspector observation was not reached
Payment state transitions
m committed event replay
0.00Required SB7.2 visual observation was not reached
Token rates
Measured by the engine itself, one record per completed model call: prefill rate is prompt tokens over time-to-first-token, generation rate is completion tokens over the decode window. Medians per node.
| Node | Calls | Prompt tok | Gen tok | Prefill tok/s | Gen tok/s |
|---|---|---|---|---|---|
| gemini | 1 | 972 | 5 | 798.0 | 58.8 |
| solar | 150 | 10,993,492 | 87,620 | 23780.4 | 6297.1 |
| fleet | 151 | 10,994,464 | 87,625 | 23457.7 | 6260.9 |
Graded browser recording
No recording: the app never served a page.
server_runs: process survives 5s without binding; serves_page: GET / -> None
Run details
- Model
- upstage/solar-mini4
- Engine events
- 0
- Repair rounds
- 0
- Started
- Oct 3, 2026, 01:28 AM
- Finished
- Oct 3, 2026, 01:54 AM