Documentation

Benchmark Methodology

Purpose, task, scoring model, baselines, posting — and the sb-6 and sb-7 tiers: the full specs, the exact score compositions, and every check explained.

Running the benchmarkBenchmark board

§01 · Purpose

Purpose

Existing leaderboards score models on question sets: a prompt goes in, an answer comes out, the answer is graded. This board scores what an agentic system builds. The system produces a working application; the application itself is then executed, probed over HTTP and in a real browser, and measured — on the poster's own hardware.

The task is frozen and the scorer is pinned per era (sb-5.2 is the era this page documents), so within one scorer version every entry is measured the same way: a 3-node local fleet, a single laptop and the Anthropic cloud baselines are read on one scale. Every post carries its evidence — the per-check results and screenshots of the built application — so a score can be audited, not just read.

Posting a run documents what local hardware can actually produce. The accumulated board is the reference dataset for local agentic builds: which fleets, which models and which node counts produce working software, and where they fail.

§02 · Task

The task

Every run builds the same application from the same frozen specification: vendorsync, a tool that syncs payments from a vendor API (Meridian, served locally as a fixture with its own documentation, base URL and API key) and shows them on a web page. The spec names four parts by file path; the scorer checks each by name.

PartRequirement
Vendor clientvendorsync/meridian.py — a MeridianClient class: fetch_all_payments() (oldest first), total_count(), and create_payment() with an idempotency key that makes repeat calls safe.
Local storevendorsync/store.py — SQLite persistence. upsert_many() must not create duplicates on re-sync; the last sync time is stored as RFC3339 UTC.
JSON APIvendorsync/api.py — GET /api/health, GET /api/payments (limit default 25, cap 100), GET /api/summary, POST /api/sync. Unknown paths return 404, bad parameters 400, every response is JSON.
Frontendvendorsync/web/ — three files: index.html (structure), styles.css, app.js. A payments table, a currency-formatted summary line, a sync control and the last-sync time. Raw ISO-8601 strings must not appear in the rendered page.

Constraints fixed by the spec: Python 3 standard library only for the backend (no package installs); the frontend is served by the backend as three static files with no build step and no CDN dependency; every timestamp a user sees is rendered human-readable in the user's locale.

§03 · Scoring

Scoring model

The scorer executes the built application: it starts the server, issues HTTP requests, drives a headless browser through the page, and measures response times. The model's own report of its work is not an input. The current scorer, sb-5.2, emits 60 checks across seven tiers; each check scores 0 to 1 and records a detail string. The overall score is the weighted sum of five components:

ComponentWeightWhat it measures
Core (tiers A–D)60%Mean of the four build tiers, weighted A 25 / B 30 / C 25 / D 20 inside the component.
Journey (J)15%Headless-browser checks: page load, sync trigger, rendered table contents.
Visual (V)10%Rendered-page checks: stylesheet served, data rows present, layout at 375px.
Performance (P)5%Response-time budgets measured on the running app.
Hard blocks10%Mean over the hard-block checks — a fixed subset a functioning product cannot fail. Graded as its own component.
TierNameWhat it checks
AStructurefiles, modules and interfaces named by the spec exist; the server starts and binds
BBehaviourHTTP requests against the running app; responses compared to expected values
CVendor contractvendor API usage checked against the vendor documentation: pagination, ordering, idempotent create
DFinesseoutput formats, edge cases and error shapes the spec requires
JJourneya headless browser loads the page, triggers a sync and reads the rendered table
VVisualrendered-page checks: stylesheet served, data rows present, mobile layout at 375px
PPerformanceresponse-time budgets measured against the running app

Hard blocks. A fixed subset of checks a functioning product cannot fail — for example, that the server binds and that the page renders data rows. Their mean is graded as a separate component with 10% weight, so a hard failure always moves the overall score.

Excellent flag. A run with an overall score of at least 0.90 is recorded with excellent: true. The leaderboard shows this as its own column.

§04 · Baselines

Baselines

The three baseline rows were produced by running the same frozen spec through Anthropic models on AWS Bedrock — one fresh build per model, scored by the same sb-5.2 scorer. The grading path is identical to a community run; only the builder differs. Baseline entries carry the same data depth as a posted run: per-tier means, the hard component, and all 60 per-check rows.

ModelBedrock model IDScoreWall clock
Claude Opus 5us.anthropic.claude-opus-50.97551,170 s
Claude Sonnet 5us.anthropic.claude-sonnet-50.9692410 s
Claude Haiku 4.5us.anthropic.claude-haiku-4-5-20251001-v1:00.7861318 s

§05 · Operation

Running the benchmark

Prerequisites

goose Local Edition desktop appThe build, the scoring and the posting all run from its Benchmark page.
An LM Studio fleet reachable from the machineOne or more nodes; the app builds with the fleet you configure.
A model loaded on every nodeAn unloaded node stalls the run until the model is loaded.
python3 on PATHThe built application and the scorer both run on it.
node with playwright installedRequired by the browser probes. Without playwright the J and V tiers read low: their checks score 0 with a PROBE UNAVAILABLE detail.

Run mechanics

A run proceeds through fixed phases: research → plan → contracts → build → verify → repair. The verify phase executes the app and raises findings; repair rounds address them until a round comes back clean or the round budget is exhausted. Typical wall-clock time on a 3-node fleet is 60–120 minutes; smaller fleets take longer.

While the run executes, the app's Benchmark page shows the current phase, the engine event count and the findings of each verify round. When the run completes, the panel shows the overall score, the per-tier means and the per-check results, with the option to publish.

Publishing and identity

Publishing POSTs the scored result to this site. The first benchmark run creates ~/.config/goose/benchmark/identity.json with a generated pseudonym of the form adjective-animal-4hex (for example crimson-heron-7f3a) and an install id. The handle is what the board displays; the install id is sent to group a poster's runs server-side and is never displayed.

Read a score against the baseline rows, not against 1.0: the reference cloud models score 0.9755 and lower on the same spec, and the scorer is built so that 1.0 stays out of reach. A local fleet in the 0.6–0.8 range is producing genuinely working software with defects the check rows name precisely.

Troubleshooting

SymptomCauseAction
Run does not start; fleet not foundLM Studio's server is not reachable, or a configured node is down.Start the LM Studio server on every node and confirm the addresses in the app's fleet configuration.
Run stalls earlyA node has no model loaded.Load the model on every node, then start a new run.
J and V checks read PROBE UNAVAILABLEnode or playwright is missing, so the browser probes cannot run.Install node and run npx playwright install chromium, then re-run. Probe-less runs score 0 on those checks.
Cancelled runCancel stops the engine mid-run.A cancelled run is not scored and nothing is posted. Start a new run.
Publish rejected with 429The rate limit is 5 posts per day per IP and per install id.Post again the next day.

§06 · Posting

Posting mechanics

Results are posted by the goose Local Edition desktop app; the site has no submission form. The app sends POST /api/benchmark-runs with a JSON payload: the overall score, per-tier means, all per-check rows, the score composition, the model identifier the fleet ran (required, prefilled by the app from engine truth), run metadata (start and finish times, engine event count, repair rounds), the poster identity, and up to five PNG screenshots captured by the render gate during the run.

Server validation

  • Unknown keys anywhere in the payload are rejected (400).
  • Consistency limits (422): score and tier values in [0, 1]; finish time after start time; swarm entrants (nodes ≥ 1) require engineEvents ≥ 100 and wallSecs ≥ 600.
  • Screenshots: PNG only (magic-byte check), at most 5 per post, at most 1.5 MB decoded each and 3.5 MB decoded total.
  • Rate limit: 5 posts per day per IP and per install id (429).
  • An accepted post (201) is live immediately; the board and the run's card revalidate on demand. Entries can be removed from the CMS afterwards.
goose Local Edition on GitHubPosting steps

§07 · Limitations

Limitations

Scorer versions are not comparable. A scorer version change alters checks and weights, so the leaderboard shows one scorer version at a time. The scorer selector defaults to the newest version with entries; older versions remain viewable as frozen historic boards, never mixed into the current one.

Scores below 1.0 are expected. The reference models score 0.9755 and lower on the same spec. A local fleet's score is read against those rows, not against 1.0.

Server validation is best-effort. The API enforces payload shape, consistency limits and rate limits, but it cannot verify that a payload came from a genuine run. Posts go live without manual review; entries can be removed from the CMS afterwards.

§07 · sb-6 · The task

sb-6 — the task every entrant receives

Build `vspro` — VendorSync Pro: a Python-stdlib-only backend plus zero-dependency offline frontend that syncs payments from the mock Meridian API v2, keeps them consistent through signed vendor webhooks and optimistic-concurrency writes, and gives a finance team a live money view including an interactive raw-WebGL 3D bar chart of payment activity.

The specification below is the prompt: every entrant — a cloud model in a single session or the local fleet — receives the same frozen document with the mock vendor's documentation URL, base URL and API key rendered in. Nothing else is provided. (The frozen spec is roughly 4,000 words (~27 KB, 456 lines of markdown) — the entrant also has to read the separately rendered Meridian API v2 docs at {DOCS_URL}, which the spec makes mandatory before starting.)

DeliverableRole
vspro/meridian.pyThe vendor client (MeridianClient): fetch_all_payments, get_payment, total_count, idempotent create_payment, create_batch (up to 20, per-item outcomes, input order kept), update_payment with the If-Match/412 recovery dance, register_webhook (idempotent by URL, challenge handshake); handles pagination, both Retry-After forms, 410 cursor_expired restarts, ETag/If-None-Match, and stalled connections under a max-10-second request timeout.
vspro/store.pySQLite persistence (Store): upsert_many that never duplicates on re-sync, filtered query returning (rows, total-under-filters), idempotent-and-ordered apply_event returning applied/duplicate/stale, Europe/Berlin (day x status) buckets, count, last_sync/set_last_sync.
vspro/api.pyThe HTTP JSON backend on 127.0.0.1: /api/health with live webhook counters, /api/payments (limit/offset/status/currency/sort, validation errors not empty results), /api/payments/<id>, /api/summary (per-currency only, no cross-currency total), /api/buckets (gap-free days x frozen statuses), /api/sync (reads keep answering during it), /api/payments/<id>/note (write-through with the concurrency dance, 409 on unresolved conflict), /api/payments/batch (local shape validation then create_batch, partial failure is normal), /api/webhooks/meridian, one frozen structured error envelope, and serving the static frontend.
vspro/web/index.htmlFrontend structure only: branded #app-header, #summary, the 3D viz panel, the payments table.
vspro/web/styles.cssAll styling: the intentional design system, exact status-badge colors, responsive 375-px layout.
vspro/web/app.jsPage behavior: server-driven table with Prev/Next and showing X–Y of TOTAL, clickable Date/Amount sort headers with aria-sort, custom status/currency dropdown filters exposing data-value, Sync-now with visible in-flight state, optimistic inline note editing with saving/saved/revert-on-409, loading/empty/error states, locale-human dates and exponent-correct money.
vspro/web/viz.jsThe 3D engine, nothing else: raw-WebGL bar chart on #viz3d implementing the frozen scene/camera/interaction/picking contracts, the #viz-toggle 2D fallback table, auto-fallback when WebGL is absent, and the graded window.vsdbg instrumentation.
vspro/__main__.pyEntry point: `python -m vspro --db PATH --port N` starts the API and page, then registers the webhook after the server is listening; must survive a missing database file and a vendor briefly unreachable at boot.
README.mdThe exact commands to install nothing, run the server, and sync.

The vendor

The app must consume Meridian API v2 exactly as its rendered docs prescribe, and the spec warns that rate limits, expired cursors, stalled connections, version conflicts and webhook signatures 'will defeat a client that did not read.' That means: cursor pagination with a mandatory restart on `410 cursor_expired`; honoring `Retry-After` 'in both documented forms'; `ETag`/`If-None-Match` conditional requests so a second sync is cheap; and a request timeout of at most 10 seconds on every vendor call, because the vendor 'may occasionally hold a connection open without answering' — a timed-out request is retried per the docs, and the sync must still land inside its 90-second budget when the vendor stalls once. Writes are optimistic-concurrency only: every update carries `If-Match` built from the resource `version`; a `412 Precondition Failed` means re-fetch, re-apply the fields, retry exactly once, and a second 412 surfaces as a conflict — never blind-write (the vendor answers `428 Precondition Required`, 'a bug in your client, not a retry case') and never retry a create with a fresh idempotency key. Batches of up to 20 creates are applied independently with per-item outcomes, and partial failure must not disturb the items that succeeded. Webhooks reverse the arrow — the vendor calls YOU: after the server is listening the app registers its endpoint (idempotent by URL), answers the unsigned `webhook.verify` challenge by echoing the hex, then treats every delivery as untrusted until the `Meridian-Signature: t=<unix seconds>,v1=<hex>` header verifies as HMAC-SHA256 of `"<t>.<raw request body>"` over the raw bytes; the vendor WILL send duplicates, out-of-order events, and (once) a forged signature, and the four live health counters (received/applied/ignored/rejected) are 'the ledger of how the app handled all of it.'

The 3D panel

The visualization is an interactive 3D bar chart of the (day, status) buckets rendered with raw WebGL on `<canvas id="viz3d">` (`{antialias: false, alpha: false}`, main thread, no OffscreenCanvas/Worker) — 'No three.js, no library, no exceptions; the asset budget enforces it.' Every contract is FROZEN and the grader recomputes the math independently against the API, the pixels, and the picking: bar centers at `x_i = (i − (D−1)/2) · 1.5` and `z_j = (j − 1.5) · 1.5` (statuses frozen as settled=0, pending=1, refunded=2, failed=3), 1.0 x 1.0 footprints, height `count · 0.25`, zero-count cells drawing no geometry; flat unlit colors with exact status hex tops, sides multiplied by 0.62 per channel, background `#0F172A`, nothing drawn but bars, and a static scene between inputs. The orbit camera has fully specified eye/basis/projection math (fovY 50°, near/far 0.1/200, defaults yaw 35 / pitch 27 / distance 30, pitch clamped [5, 85], distance [10, 90], yaw compared modulo 360), and interaction is pinned numerically: drag applies `yaw ← yaw − 0.35·Δx`, `pitch ← clamp(pitch + 0.35·Δy, 5, 85)`; wheel applies `distance ← clamp(distance · exp(0.0012 · deltaY), 10, 90)` without scrolling the page; double-click resets. Hover shows tooltip `#viz-tooltip` within 150 ms with `<count> <status> · <day>`; clicking a bar drives `#status-filter` and refreshes the table; picking is 'geometric truth' — the nearest rendered surface at that CSS pixel, occlusion decided exactly as the depth buffer says. A `#viz-toggle` button swaps to a real 2D fallback `<table id="viz-fallback">` of the same buckets, shown automatically (with a visible in-panel notice, no throw) when WebGL is unavailable. The page must expose `window.vsdbg` with `version: 3` and truthful `scene()`, `camera()`, `setCamera()`, `project()`, `pick()`, and `frames()` — the grader cross-checks all of it against screenshots, and 'an instrumentation layer that reports a scene the canvas does not show scores as broken, not as clever.'

The data is hostile

The vendor fixture is 1,553 payments spanning 14 Europe/Berlin calendar days that deliberately cross the 2026-03-29 DST transition, in 4 currencies (EUR, USD, JPY, KWD) and 4 statuses (settled, pending, refunded, failed). Two classic shortcuts are graded as wrong on purpose. First, money: amounts are integers in minor units end to end, and each currency has its own exponent — EUR/USD 2, JPY 0, KWD 3 — so minor units are not a common denomination and 'summing minor units across currencies is meaningless and forbidden'; there is no cross-currency total anywhere in the API or UI, and rendering must respect the exponent (`129900 EUR → €1,299.00`, `129900 JPY → ¥129,900`, `129900 KWD → KWD 129.900` — 'a yen amount with two decimals, or a dinar truncated to two, is wrong money, and money is the product'). Second, time: bucketing happens on the INSTANT, assigned to its Berlin calendar date — not the raw string's date and not the UTC date. Because Berlin jumps from UTC+1 to UTC+2 mid-fixture, payments near midnight land on different calendar days under UTC than under Berlin time, so 'UTC-day bucketing produces measurably wrong counts' — a detector the grader can read straight out of `/api/buckets`.

Boot contract

The documented boot line is `python -m vspro --db PATH --port N` — it 'starts the backend serving the API and the page, then — after the server is listening — registers the webhook with the vendor.' The scorer launches the entrant's build with exactly this line, pointing it at its own database path and port, and everything downstream — API probes, webhook traffic, the browser session, every latency budget — is measured against the process this one line starts. The spec pins the two boot failure modes explicitly: the process 'must not crash when the database file does not yet exist, and must start (serving whatever is already local) even if the vendor is briefly unreachable at boot' — an app that dies on either never gets scored on anything else.

Performance budgets, as specified

  • First data rows rendered within 2 seconds of page load.
  • The 3D canvas shows its first non-background frame within 3 seconds of page load.
  • `GET /api/payments` at `limit=50` answers in under 150 ms at p95 — including while a sync is running with 8 concurrent readers.
  • `GET /api/buckets` answers in under 200 ms at p95.
  • `GET /api/summary` answers in under 150 ms at p95.
  • `POST /api/sync` completes the full fixture within 90 seconds, documented waits included.
  • During a scripted drag, the scene keeps up with the pointer: `vsdbg.frames()` advances by at least 0.8 frames per pointer move event delivered — the scene is event-driven, so each move should draw; dropping more than one in five is lag — and a camera change is visible on the canvas within 250 ms of the input that caused it.
  • The hover tooltip appears within 150 ms.
  • An optimistic note edit paints the new value within 100 ms of confirm — before the network responds.
  • `index.html` + `styles.css` + `app.js` + `viz.js` total at most 150 KB uncompressed.

The spec demands the page be built 'as a product, not as a debug view over the API': an intentional visual design with a real palette of strong solid accent colors, a clear typographic hierarchy, and a branded header bar carrying the app name. Three prohibitions are explicit: never faded pastel washes (saturated solid colors over tints), never a left accent line or rail decorating cards or rows, and never browser-native controls where custom styling is expected — no default `<select>`, no `alert()`/`confirm()`/`prompt()`; filters are custom dropdowns and note editing is a custom inline editor with a non-blocking `#notice` (`role="status"`). Status badges use the four frozen hex values shared with the 3D chart (`settled` #16A34A, `pending` #F59E0B, `refunded` #8B5CF6, `failed` #DC2626), distinct in computed color, not only in text. Every user-visible timestamp renders human-readable in the user's locale — a raw ISO-8601 string with an offset must never appear. The page handles loading, empty (with a call to sync) and error states visibly and distinctly, the viz panel owns `#viz-empty` and `#viz-error` ('never a blank panel, never a spinner that never resolves'), and at a 375-px viewport it lays out cleanly with no horizontal scroll, the canvas full-width at min height 240 px and still interactive.

§08 · sb-6 · Composition

The number, exactly

score = 0.88 × core + 0.12 × gate × excellence

score = 0.88 * inner + 0.12 * gate_fraction * e_mean. `inner` is the weighted sum of the nine non-E tier means (A 0.06, B 0.12, C 0.12, D 0.10, J 0.12, V 0.08, P 0.06, T 0.14, HARD 0.20 — weights sum to exactly 1.0, and the scorer asserts T+HARD >= 0.34 and E >= 0.10 so the hard axis and the excellence slice cannot be quietly compressed). Each tier mean averages the gamma-transformed scores of that tier's weight-carrying, measured checks: score^gamma_hard for T and HARD, score^gamma_core for the rest (both default 1.0 until calibration, hard-capped at 4.0), and k_P tightens only the TOP rung of each P-tier latency ladder (top budget / k_P; the printed spec budgets stay as written). The E (excellence) tier is a separate 0.12 slice: it is multiplied by gate_fraction (how many named excellence conditions are perfect) and e_mean (the mean of the E-tier checks themselves), so a perfect app reaches exactly 1.0 and everyone else earns the slice in proportion. gamma, k_P, and several rung ladders load from sb6-thresholds.json, which is pinned: a thresholds file claiming calibrated=true whose sha256 does not match the pin baked into the scorer makes it REFUSE to score at all, and until the calibration fit is frozen every report carries an UNCALIBRATED banner and the version string sb-6.0-rc. NEW (severity model, 2026-08-19): the composed score is then multiplied by a CRITICAL-DEFECT factor — seven checks whose failure means crash, wrong money, data loss, or a dead primary flow (server_runs, sync_completeness, b_summary_currency, b_buckets_dst, h_durability, j_loads_data, j_sync_journey) each contribute factor = m + (1-m)*check_score with m = 0.6 (calibration-owned), compounding. A clean run multiplies by exactly 1.0; a fully failed critical alone costs 40% of everything. Wrong money and data loss are additionally CLIFFS at the check level (a cross-currency sum or any row lost across kill+reboot scores 0.0, never a floor). The freeze gate runs a monotonicity selftest: synthetic single-defect runs must order wrong-money < data-loss < dead-flow < console-error < minor < cosmetic, or the scorer refuses to freeze.

Tier weights — the core's composition

A
B
C
D
J
V
P
T
HARD

A Structure · 0.06 — The named files, classes and endpoints exist and the server boots.

B Behaviour · 0.12 — What the API actually returns on the wire — sync completeness, shapes, ordering, money math.

C Vendor contract · 0.12 — How the app consumes the vendor: cursor paging, traps, batch partial failure, persistence.

D Finesse · 0.10 — Engineering quality — timeouts, atomic writes, indexes, content types, error affordances.

J Journeys · 0.12 — Real user flows in a real browser: first use, the sync click, error and empty states.

V Rendered truth · 0.08 — What the page visibly shows: money, dates, statuses, filtering, pagination, 375px.

P Performance · 0.06 — The spec's latency and interactivity budgets, measured p95 under real requests.

T 3D panel · 0.14 — The raw-WebGL bucket chart: real draws, correct scene math, camera, picking, fallback.

HARD Hard mechanisms · 0.20 — Durability, sync discipline, the webhook ledger, the If-Match conflict dance.

E Excellence · 0.12 — The last 12%: frames under drag, latency under load, optimistic paint — unlocked per perfection condition.

The excellence slice — 14 perfection conditions

The last 12% unlocks in proportion to named perfection conditions, then pays out at the excellence tier's own measured mean. Every run page shows which conditions that run met.

  • j_first_use == 1.0 — first load must be fast, reconcile the rendered total with the fixture truth, and be console-clean; first use is the one experience every real user has.
  • j_sync_journey == 1.0 — the headline Sync flow must work end to end in a real browser (button found, in-flight state, completion, view visibly refreshed against the half-seeded vendor).
  • j_error_state == 1.0 — when the backend fails, the user must see an actionable error (with a working retry when the probe exercised one), not a blank page.
  • j_empty_state == 1.0 — a fresh install on an empty db must explain itself (empty-state text, working Sync CTA), never render phantom data or a blank page.
  • console_clean: exactly 0 console errors across the NOMINAL scenarios only (load, sync, empty, viz) — error and viz-fallback scenarios produce expected network noise and are excluded; if any nominal probe was unavailable the condition is unproven and honestly fails rather than passing vacuously.
  • v_responsive_375 >= 1.0 — the UI must actually work at 375px; excellence includes the phone viewport, not just the dev's monitor.
  • v_dates_readable >= 1.0 — rendered dates must be human-readable; raw ISO strings in cells are a tell that no one looked at the page.
  • t_scene_binding >= 1.0 — the 3D scene must be bound to the real data, not a decorative canvas.
  • p_list_latency == 1.0 — the payments list endpoint meets its top latency rung.
  • p_buckets_latency == 1.0 — the buckets endpoint (the 3D chart's data source) meets its top rung.
  • p_summary_latency == 1.0 — the summary endpoint meets its top rung.
  • p_page_interactive == 1.0 — the page reaches interactive within the top budget.
  • p_first_frame == 1.0 — the first 3D frame draws within the top budget.
  • p_sync_wall == 1.0 — a full sync completes within the top wall-clock budget. WHY PROPORTIONAL: the original all-or-nothing gate measured as a dead zone — 7 of 7 real entrants had it locked (one 0.75 journey rung erased a measured-perfect E tier), silently compressing the whole board by x0.88 so no run page could reconcile its rows with its score. Each condition now unlocks its share of the 0.12 slice: still harsh (full credit requires ALL of them perfect, and the E checks themselves must also measure well), never a cliff.

One defect, not nine

One root defect attributes its downstream failures so the report reads as one defect, not many. If sync_completeness scores below 1.0 (any shortfall, not only zero), its eight dependents — payment_row_shape, total_field, chronological_order, b_summary_currency, summary_bounds_utc, b_buckets_dst, row_integrity, h_sync_discipline — are flagged as downstream of that one root wherever they also fell short. This is attribution ONLY: no score changes, but the verdict says loudly 'ONE defect, not 9', so an entrant whose sync lost rows is not read as having nine independent failures.

Diagnostic rows

Nine checks are weight-zero diagnostics: resync_idempotent, update_propagation, restart_persistence, concurrent_sync_safe, store_atomic_upsert, j_loads_data, j_console_clean, local_pagination, input_validation. They are computed and printed on every report (marked with a dot) but excluded from every tier mean, because their measurements are absorbed by compound checks (c_api_depth = gate x min(schema, pagination, validation); j_first_use = gate x min(first-data ladder, reconciliation, console-clean)). Counting them twice would let the same evidence stack credit — or stack punishment.

Probe failures never blame the app

Harness-attributable absence is never the app's fault. When a probe errors, a vendor-mock surface is missing, a viz section is lost to the probe's hard timeout, or an under-load measurement cannot prove the readers actually overlapped the sync, the row is marked PROBE UNAVAILABLE: it is EXCLUDED from its tier mean (the mean is over what was actually measured), listed loudly in the verdict, and never converted into an app zero — an unproven measurement licenses nothing, in either direction. App-attributable absence (no canvas, no rows, no debug surface) still scores 0. Any unavailable row also disqualifies the run from the 'excellent' flag and blocks a reference freeze.

Worked example

A worked example under the severity model: inner 0.9875, gate fraction 13/14, e_mean 1.0 composes to 0.88 x 0.9875 + 0.12 x 0.9286 x 1.0 = 0.9804. If the run also summed money across currencies (b_summary_currency 0.0, a critical cliff), the critical factor is 0.6 + 0.4 x 0.0 = 0.6, and the published score is 0.9804 x 0.6 = 0.5882 — wrong money craters the number, as it should.

The ruler is itself gated: Before any calibration threshold is trusted, the hand-written golden reference app must pass the freeze gate (--reference). The scorer grades the reference tree and REFUSES to emit freeze-marked output (exit 3, no sb6-reference-pass.json) if ANY non-calibration-owned, non-diagnostic check scores below 0.95, if any check was probe-unavailable, or if the excellence gate is not fully open — on the principle that a bar the reference itself cannot clear is a harness defect, not an app defect, until proven otherwise. The ten calibration-owned checks (drag frames, under-load latency, optimistic paint, tooltip, and the six P-tier budgets) are exempt from the 0.95 sweep because their cut points are SET from the reference's own distribution — grading the reference against them pre-freeze would be circular. On pass, a marker records the scorer version, score, date, and the sha256 of the thresholds file, and only then may thresholds be fitted.

Fairness: Scoring is serial — one tree, one vendor mock, one app process per invocation, so no entrant's measurements contend with another's. It is hermetic — every scoring run deletes any graded db (and WAL/SHM sidecars) left by a previous run before booting the app, because a warm leftover db turns the first sync into a re-sync and corrupts update-propagation, webhook-counter, conflict-dance and ETag measurements (a golden re-gate measured 1.00 -> 0.7995 from this alone); every invocation starts from an empty db exactly like the first one did. And the vendor mock is served at the port the entrant's spec advertised, with the app told where via MERIDIAN_BASE_URL — the entrant is graded against the environment its own spec asked for, never a scorer-convenient one.

§09 · sb-6 · Every check

Every check, piece by piece

All 69 checks the sb-6 scorer runs, grouped by tier. For each: what it measures, the instrument that measures it, what a 1.0 requires, and exactly what lowers it. This documentation is generated from the scorer's source and verified against it — when a run page shows a check at 0.4, the rung that produced 0.4 is written here.

AStructureweight 0.06 · 7 checks

CheckWhat it measures · howEarns 1.0Loses points
a_asset_budgetThe whole frontend fits the 150 KB source budget — the budget exists so a vendored 3D library cannot.stat().st_size sum over the four web files on disk (index.html, styles.css, app.js, viz.js); missing files contribute 0 bytes.Combined size of the web files present is at most 150 KB (153,600 bytes).0.3 if total is over budget but within 120% (≤180 KB); 0.0 beyond that; 0.0 if no web files exist at all.
health_shape/api/health exposes the documented monitoring shape: status, payments, last_sync, webhook at top level plus the 5 webhook counters (registered, received, applied, ignored, rejected).Key-presence inspection of the JSON body captured from the boot-time GET /api/health.All 4 top-level keys and all 5 webhook sub-keys present (9/9).Linear: score = keys_present/9, so each absent key costs 1/9; a missing or non-dict webhook object forfeits all 5 counter slots.
interfaces_declaredThe two named classes declare the documented interface: MeridianClient with 7 methods (fetch_all_payments, get_payment, total_count, create_payment, create_batch, update_payment, register_webhook) and Store with 8 (upsert_many, query, get, apply_event, buckets, count, last_sync, set_last_sync).Static analysis: ast.parse of meridian.py and store.py, collecting FunctionDef/AsyncFunctionDef names inside each ClassDef — no import, no execution; free functions or differently-named classes do not count.All 15 methods declared under exactly those class and method names.Linear: score = matched/15, so each missing or misnamed method costs 1/15; a file that fails to parse contributes zero methods for its class.
modules_presentThe submitted tree contains every file the spec names by path: meridian.py, store.py, api.py, __main__.py, web/index.html, web/styles.css, web/app.js, web/viz.js inside the vspro/ package.Filesystem stat (Path.is_file) on each of the 8 spec-named paths in the submitted tree; a one-level-nested tree containing vspro/ is auto-detected first.All 8 files exist at their exact spec paths.Linear: score = present/8, so each missing file costs 0.125; a file at the wrong path or wrong name counts as missing.
server_runsThe app boots as a real process and reports healthy over HTTP.Spawns `python -m vspro --db <fresh> --port <free>` as a subprocess with MERIDIAN_BASE_URL pointed at the vendor mock, then polls GET /api/health every 0.4 s for up to 25 s./api/health returns 200 with a JSON body within 5 s of spawn.0.75 if healthy but slower than 5 s (within the 25 s window); 0.4 if the port answered but health never returned 200; 0.15 if the process survives 5 s without binding the port; 0.0 on crash at boot (last 800 bytes of output recorded as the boot error).
serves_pageThe backend itself serves the frontend: GET / returns the page and the three assets come back with correct content types, as a real browser would need.HTTP GETs against the running server: / must return 200 containing markup, then styles.css, app.js, viz.js are fetched (bare and web/-prefixed paths tried); each asset passes when its file on disk is non-empty AND the live response's Content-Type contains 'css' or 'javascript' respectively.GET / is 200 with markup and all 3 assets exist non-empty on disk and are served with correct content-type families.0.7 with 2/3 assets correct; 0.4 with 0-1/3 (page still 200 with markup); 0.2 when web/index.html exists on disk and a server answered / but the response was not a 200 with markup (a 200 with an empty or non-HTML body lands here too); 0.0 when no server answered at all.
sync_shapePOST /api/sync returns the documented result envelope so a caller can tell what a sync did: fetched, inserted, updated, total.Key-presence inspection of the JSON body from the scorer's first POST /api/sync (run against the half-capped vendor collection; only keys are graded here, values are graded in tiers B/HARD).All 4 documented keys present in the response.Linear: score = keys_present/4, so each missing key costs 0.25; a failed or non-JSON sync response scores 0.

BBehaviourweight 0.12 · 14 checks

CheckWhat it measures · howEarns 1.0Loses points
b_buckets_dstGET /api/buckets — the 3D chart's data source — buckets payments into Europe/Berlin calendar days per status, correct across the DST switch (the deliberate data-correctness trap).HTTP GET /api/buckets; every (day, status) → count cell is compared against the fixture's EXPECTED_BUCKETS, cell-set size against the expected shape, and the declared timezone field against "Europe/Berlin".All cells exact, cell count matches the expected shape, and timezone == "Europe/Berlin" (weighted 0.7 exact-cells + 0.2 shape + 0.1 timezone).Exact-cell fraction scales the 0.7 slice (an off-by-one on the DST day costs its cells); a wrong cell count halves the shape slice to 0.1; a missing or different timezone declaration loses 0.1; no cells at all scores 0; missing fixtures make the row PROBE UNAVAILABLE.
b_error_envelopeError responses carry the documented structured envelope — error.code and error.message strings, plus field_errors[] with per-field path and code where field-level detail is expected.HTTP validation matrix against the running app (limit=-1, limit=abc, offset=-5, sort=bogus, status=bogus, unknown route, and an invalid batch item whose field path must reference items[...]); this check grades only the envelope SHAPE — the status-code half of the same responses is credited once, in c_api_depth (anti-stacking split).Every observed error response has a well-formed envelope (0.6 weight) and every field-error-expecting case returns field_errors entries with string path and code — the batch case specifically an "items[" path (0.4 weight).Each half scales by its fraction of passing cases; an envelope without field_errors on validation cases caps at 0.6; no error responses observed at all scores 0.
b_summary_currencyGET /api/summary reports exact per-currency (count, total_minor) pairs and never sums money across currencies — a cross-currency sum is wrong money.HTTP GET /api/summary; each of the 4 currency buckets (EUR/USD/JPY/KWD) is compared against the fixture's EXPECTED_BY_CURRENCY, and every top-level integer field is scanned for a value equal to the actual cross-currency minor-unit sum (F16: the cap fires only on a real money sum, never on a suspicious key name).4/4 buckets exact on both count and total_minor, no field carrying the cross-currency sum, and by_currency sorted by currency code.Base score is k/4 exact buckets; any field equal to the cross-currency sum (and not a legitimate single-currency total) caps the score at 0.25; an unsorted by_currency array multiplies the result by 0.9; missing fixtures make the row PROBE UNAVAILABLE.
chronological_orderThe default payments page is ordered by created_at as a parsed instant, not as a string — mixed UTC offsets make string ordering wrong.HTTP GET /api/payments; every created_at is parsed as RFC3339 (Z normalized to +00:00) to an epoch instant, and adjacent row pairs are compared numerically.Every adjacent pair is non-decreasing by instant.Score is the fraction of ordered adjacent pairs; a single unparseable created_at anywhere on the page scores the whole check 0 (timestamps are not RFC3339); fewer than 2 rows scores 0.
input_validationdiagInvalid input is rejected with the documented status code instead of being accepted or crashing. Weight-zero diagnostic — the status-code credit is carried by c_api_depth's matrix, and this row exists for attribution.HTTP validation matrix on the running app: /api/payments with limit=-1, limit=abc, offset=-5, sort=bogus, status=bogus and a POST /api/payments/batch with an invalid item must all return 400; the unknown route /api/nope must return 404.All 7 matrix cells return exactly the documented status code.Score is the fraction of correct status codes; a matrix that was never exercised scores 0. Score does not enter the Tier B mean (DIAGNOSTIC set).
json_everywhereEvery API response — success paths and error paths alike — is parseable JSON labeled application/json; an HTML error page mid-API breaks every client that trusted the contract.HTTP sample of six endpoints on the running app (/api/health, /api/payments, /api/summary, /api/buckets, the 404 route /api/nope, and the 400 case /api/payments?limit=-1), grading each response's raw body for JSON parseability and its Content-Type header for "json".All six responses both parse as JSON (0.5 each) and carry a json Content-Type (0.5 each).Each response contributes its halves independently — a JSON body served as text/html keeps 0.5 of that response's credit; error routes count exactly like success routes; no responses sampled scores 0.
local_paginationdiagThe local API honors its own paging contract: default page of 50 rows, limit clamped to a 200 cap, and limit/offset actually applied. Weight-zero diagnostic — the behavior is absorbed by compound checks, and this row exists for attribution.Three HTTP GETs against the running app: /api/payments (default), /api/payments?limit=500 (over-cap), and /api/payments?limit=5&offset=5 (windowed).All three parts pass: default data[] length == 50, the over-cap response echoes limit == 200 with at most 200 rows, and the windowed request returns exactly 5 rows.Each failing part costs a third. Score does not enter the Tier B mean (DIAGNOSTIC set).
payment_row_shapeA payment row on the wire carries exactly the 10 documented keys (id, amount_minor, currency, created_at, settled_at, status, version, note, counterparty_name, country).HTTP GET /api/payments against the running app; the scorer takes the key set of the first row of data[] and compares it with the frozen v3 key vocabulary.All 10 documented keys present and no undocumented extras.Score is the fraction of the 10 documented keys present; any extra key beyond the documented set subtracts a flat 0.2 (the spec says "exactly the keys"); an empty data[] scores 0.
resync_idempotentdiagRunning sync a second time is safe: it inserts nothing and the ledger total is unchanged — a non-idempotent re-run duplicates rows and inflates the ledger. Weight-zero diagnostic absorbed by the compound sync checks.The harness POSTs /api/sync a second time after the full sync completes and reads the inserted and total counters from the response body.Second sync reports inserted == 0 and total == 1553.total correct but inserted != 0 scores 0.5; wrong total scores 0; a failed second sync scores 0; missing fixtures make the row PROBE UNAVAILABLE. Score does not enter the Tier B mean (DIAGNOSTIC set).
row_integrityEvery row of the entire collection is well-formed on the wire: documented keys, integer minor-unit amounts (never floats), parseable timestamps, and full id coverage — malformed rows poison every downstream consumer.The harness pages GET /api/payments?limit=200 across all 1553 rows and re-parses the raw response bytes with a json parse_float sentinel, so a float amount is detected as a property of the bytes on the wire, not of Python's numeric view after decoding.All rows carry at least the 10 documented keys (0.35), every amount_minor is an int with zero floats anywhere in any payload (0.25), every created_at is a string of at least 19 chars (0.15), and distinct ids cover all 1553 fixture payments (0.25).Each slice scales by its per-row fraction; a single float observed on the wire zeroes the entire 0.25 integer-amounts slice regardless of how many rows were clean; no rows on the wire scores 0; missing fixtures make the row PROBE UNAVAILABLE.
summary_bounds_utcGET /api/summary states the period it covers with oldest/newest bounds that exist, are UTC-stamped, and are ordered — local-offset or missing bounds misstate coverage.HTTP GET /api/summary; the oldest and newest fields are checked for presence, for a Z or +00:00 suffix, and for oldest < newest after RFC3339 parsing to instants.All three components: both bounds present (0.4) + both UTC-suffixed (0.3) + correctly ordered as instants (0.3).Each component drops its own slice independently: missing either bound loses 0.4 and forfeits the other two, a local-offset stamp loses the 0.3 UTC slice, and oldest ≥ newest (or unparseable) loses the 0.3 ordering slice.
sync_completenessAfter a full sync the app's local store holds every one of the fixture's 1553 vendor payments — an incomplete sync silently loses money rows.The harness half-seeds the vendor mock (visible cap), POSTs /api/sync directly, clears the cap, then the headless-Chromium probe clicks the page's Sync button (#sync-now or any visible control whose accessible name matches /sync/i) against the full collection; if a follow-up GET /api/health shows a row count different from 1553 (missing or duplicated rows alike) the harness runs a fill sync so downstream checks stay real, and the scorer grades the best-evidenced full count among post-restart /api/health payments, the fill sync's total, the second sync's total, and GET /api/payments total.Any of those four evidenced counts equals EXPECTED_TOTAL (1553).Score is linear: min(best_count/1553, 1.0) — 1400 rows scores 0.90. Missing fixture constants make the row PROBE UNAVAILABLE (excluded from the tier mean, never converted to an app zero). Any shortfall (<1.0) also attributes eight dependent checks (payment_row_shape, total_field, chronological_order, b_summary_currency, summary_bounds_utc, b_buckets_dst, row_integrity, h_sync_discipline) to this root cause in the verdict — attribution only, no score change.
total_fieldGET /api/payments reports the correct collection total — a wrong total breaks every caller's paging math.HTTP GET /api/payments; the returned total integer is compared against the fixture's EXPECTED_TOTAL (1553).total == 1553 exactly.Off by at most 2 scores 0.5; within 20% of 1553 scores 0.25; anything else (including a non-integer) scores 0. Missing fixture constants make the row PROBE UNAVAILABLE.
ui_offlineThe frontend runs with zero external code — no CDN scripts, stylesheets, or remote imports — because the spec requires fully offline operation.Static regex analysis over the page as actually served: index.html plus the three assets (styles.css, app.js, viz.js) fetched over HTTP from the running app, scanned for src=/href= references to http(s) URLs and dynamic import("http…") calls.Zero external asset references across the served page and all three assets.Binary: a single external reference scores 0 (the count is reported in the detail); no page to inspect scores 0.

CVendor contractweight 0.12 · 6 checks

CheckWhat it measures · howEarns 1.0Loses points
c_api_depthThe app's local API contract in depth — exact row schema, pagination semantics, and input validation — as a compound where the weakest facet bounds the whole.HTTP probes against the running app: walks /api/payments in 200-row pages and checks every returned row's key set equals the 10-key payment schema (id, amount_minor, currency, created_at, settled_at, status, version, note, counterparty_name, country); five pagination cells (default page is exactly 50 rows; limit=500 is capped to 200 and echoed; limit=5&offset=5 returns exactly 5; sort=-created_at puts the true max created_at first; status=refunded returns only refunded rows with an integer total); and a validation matrix (limit=-1, limit=abc, offset=-5, sort=bogus, status=bogus must return 400, /api/nope must return 404, plus an invalid batch item must return 400).The rows_on_wire gate holds (at least one row came back from the paged walk) and min(schema fraction, pagination fraction, validation fraction) = 1.0 — every row exactly the documented 10 keys, all five pagination cells correct, every status code in the matrix correct.Compound gate × min: no rows on the wire scores 0 outright; otherwise the score IS the weakest of the three fractions, so a perfect schema cannot buy back broken validation. Extra or missing row keys, any failed pagination cell, or any wrong status code drags its fraction down. Absorbs local_pagination and input_validation (their B-tier rows are weight-zero diagnostics).
c_batch_partialPOST /api/payments/batch treats a partial failure as a normal outcome: input order preserved, the failing item reported with the vendor's error code, nothing rolled back, and no retry storm against the vendor.POSTs a scripted 3-item batch to the app in which item 2 exceeds the vendor's per-payment amount limit (AMOUNT_LIMIT_MINOR = 5,000,000 minor units), then reads two instruments: the app's batch response body, and the vendor mock's create-operation ledger (batch_trace()), which counts duplicate creates, retries of a failed item under the same idempotency key, and fresh-key resubmissions of the same content.All four quarter-credits: results[] carries index 0..n matching input order; the vendor ledger shows zero duplicate creates (each good item created exactly once); the failed item is reported with error code amount_over_limit; and the ledger shows zero retries of the failed item — neither same-key retries nor same-content fresh-key retries.Each failed part costs 0.25 of the score (equal quarters). A rollback (good items missing), a duplicated create, a swallowed error code, or any retry of the doomed item each burn their quarter. Both the response and the vendor ledger absent means the batch exercise never ran — PROBE UNAVAILABLE, not an app zero.
restart_persistencediagSynced rows survive a hard kill and reboot of the app process — the SQLite file is a real store, not decoration.Kill+reboot probe: reads the payments count from /api/health, SIGKILLs the app's entire process group (no graceful shutdown), respawns the app on the same database file and port, and polls /api/health for up to 20 seconds to read the count again.Every row survives: rows_after_restart / rows_before_kill ≥ 1.0.Score is the surviving fraction of rows. If the app never becomes healthy again after the SIGKILL, the score is 0 ("a crash loses the ledger"). If there were no rows before the kill there is nothing to prove and the score is 0. Weight-zero diagnostic: absorbed as the 'persists' component of h_durability.
update_propagationdiagVendor-side status changes become visible in the app's local data after a resync — an app that only ever INSERTs shows stale statuses forever.The harness calls the vendor mock's mutate_statuses(), which flips 12 spread payments (pay_0100 … pay_1200) to new statuses with version bumps and returns the exact {id: new_status} map; it then POSTs /api/sync (the graded sync #3) and GETs each mutated /api/payments/{id} from the app, comparing the local status field to the mock's declared target per id.All 12 mutated payments show their new status locally after sync #3 (score = seen/changed, capped at 1.0). The per-id comparison is deliberate: a bulk count of the target status would be vacuously passable on a fixture that already contains rows in that state.Score is the fraction of mutated ids whose new status is visible locally. If the mutation pass never ran or sync #3 is absent, the score is 0. Weight-zero diagnostic: the identical measurement is graded inside h_sync_discipline as its 'propagates' component, so this row informs without double-counting.
vendor_cursor_pagingThe client walks the vendor's payment collection using only the documented cursor protocol — no offset-guessing or invented paging parameters.Parses the vendor mock's JSONL request trace (every request the app ever made to the mock), selects GET requests to the documented list path (/v2/payments), and counts requests carrying a cursor query parameter versus requests carrying any undocumented paging parameter (offset, page, start, or skip).At least one list request used a cursor and zero list requests carried offset/page/start/skip. Cursor-less walk-starts are free — the score is 1 − offenders/list_requests, so a clean cursored walk is 1.0 regardless of how many walks the harness triggered.Each list request carrying an undocumented paging param subtracts 1/list_requests. Zero cursored requests scores 0 ("the collection cannot be walked without the documented cursor"), as does making no list requests at all. A missing vendor trace is PROBE UNAVAILABLE — excluded from the tier mean, never converted to an app zero.
vendor_trapsThe client survives the vendor's three documented trap behaviours: 429 rate-limiting with Retry-After in both forms (seconds and HTTP-date), a 410 cursor_expired that demands a cursor-less restart, and a one-shot 12-second hold on the final sync page.Reads the vendor mock's own trap ledger (trap_results()), which grades every trap instance in its request trace: after each 429 the next list request must arrive no earlier than the advertised Retry-After (0.3 s tolerance for the seconds form, 1.3 s for HTTP-date's 1-second wire granularity); after each 410 the next list request must carry no cursor; for the stall, a retry of the held page must land while the 12 s hold is still open (client-socket drop kept as corroboration) and the page must complete afterwards. Each trap fraction is the MIN over all its instances; the ledger blocks until an in-flight stall resolves its verdict.Mean of the three ledger fractions equals 1.0: every 429 was waited out to the advertised time in both header forms, every expired cursor was answered by restarting the walk without a cursor, and the stalled final page was timed out client-side and retried during the hold.Score is the arithmetic mean of retry_after, cursor_expiry, and stall, each in [0,1]. Waiting the full 12 s hold out instead of timing out earns stall = 0.5. One early retry after any 429, or one retry reusing a dead cursor, zeroes that trap entirely (min over instances). An empty ledger means the app never exercised the vendor's trap surfaces (usually a dead server) and is PROBE UNAVAILABLE.

DFinesseweight 0.10 · 7 checks

CheckWhat it measures · howEarns 1.0Loses points
api_content_typeEvery API response — success and error paths alike — declares a JSON content type, so strict clients do not break.Captures the Content-Type header on six live GETs: /api/health, /api/payments, /api/summary, /api/buckets, plus the error routes /api/nope (404) and /api/payments?limit=-1 (400).All 6 responses carry a Content-Type containing 'json' (case-insensitive) — including the 404 and the 400.Linear: each non-JSON-typed response costs 1/6; an HTML error page on the 404/400 routes is the typical deduction; 0.0 if the server never answered so nothing was sampled.
client_timeoutsThe vendor client bounds its requests so one unresponsive vendor call cannot hang the sync forever.Behaviour first: the vendor mock arms a one-shot stalled response (arm_stall) during the under-load phase and its trap ledger (trap_results) records whether the app's sync still returned within bound; if not proven, a regex grep of meridian.py for `timeout=` is the residual.The trap ledger's stall entry is 1.0 — the sync demonstrably survived the vendor stall within bound.0.4 residual if the stall was not survived (or the ledger is empty) but `timeout=` appears in meridian.py source; 0.0 with neither. The source grep is deliberately demoted to a residual — behaviour outranks the grep that once decided model rankings.
concurrent_sync_safediagTwo operators pressing Sync at once neither duplicate nor lose rows. Weight-zero diagnostic: the same measurement is the `concurrent` component inside the HARD-tier h_durability compound.Fires two POST /api/sync requests from two threads simultaneously (joined with a 240 s timeout), then reads the payments count from GET /api/health and compares it to the fixture's EXPECTED_TOTAL.The post-race payments count equals EXPECTED_TOTAL exactly.Continuous: score = 1 − |count − EXPECTED_TOTAL| / EXPECTED_TOTAL, so both duplicated and lost rows cost proportionally; 0.0 if the count was never observed. Fixture constants missing is a PROBE UNAVAILABLE refusal (excluded from the tier mean), never an app zero.
store_atomic_upsertdiagWrites merge atomically at the SQL level, so overlapping syncs cannot duplicate rows via read-then-write races. Weight-zero diagnostic: the same evidence is the `atomic` component inside the HARD-tier h_durability compound.Static analysis: regex extracts the INSERT ... INTO payments statement from store.py (whole file as fallback) and classifies the write pattern.ON CONFLICT ... DO UPDATE inside the extracted insert statement.0.5 for INSERT OR REPLACE / INSERT OR IGNORE / REPLACE INTO (writes but does not merge); 0.3 for a select-then-insert pattern (SELECT ... WHERE ... id plus INSERT INTO — racy); 0.0 with no upsert found or no store.py.
store_indexedThe SQLite payments table is keyed, so upserts are not full table scans.Static analysis: case-insensitive regex over store.py source for PRIMARY KEY, UNIQUE, or CREATE [UNIQUE] INDEX.Any of the three keying constructs appears in store.py.Binary: 0.0 if none appear or store.py is absent — there are no partial rungs.
ui_error_actionableWhen the backend is unreachable, the UI shows an error the user can act on rather than a blank or mute page.Runs the product probe's error scenario in headless Chromium (Playwright) with --block-api: a route interceptor aborts every /api, /data, /graphql and *.json request with connectionrefused while the document loads, waits for network idle + 1 s, then scans visible leaf elements — excluding table cells, buttons, legends, filter chips and bare status words like 'failed' — for error phrasing, emitting errorStateVisible and the matched actionableText.A visible error message whose text matches try again / retry / check / running / refresh. (If the probe ever emits retryRecovered, a working retry is required for 1.0 and actionable-text-with-dead-retry drops to 0.6; the current probe does not emit it, so the text rung is authoritative.)0.3 for a visible but generic error indication with no actionable phrasing; 0.0 with no error affordance at all. A probe failure is scored PROBE UNAVAILABLE (excluded from the tier mean), never converted into an app zero.
uses_max_limitThe client requests the vendor's documented maximum page size (100) instead of paying avoidable round trips.Reads the vendor mock's JSONL wire trace of every request the app made and takes the largest integer `limit` query parameter across all GET list-endpoint requests.At least one vendor list request with limit=100 (or above).Proportional: score = largest_limit/100, so limit=50 earns 0.5; 0.0 if the client never sent an explicit limit at all.

JJourneysweight 0.12 · 6 checks

CheckWhat it measures · howEarns 1.0Loses points
j_console_cleandiagWhether normal use of the app (loading it and running a sync) produces JavaScript console errors or uncaught page errors.Playwright console and pageerror listeners collect error events across both the load scenario and the sync scenario in headless Chromium; the scorer counts the union.Zero console errors and zero uncaught exceptions across both browser sessions.Exactly one error scores 0.5; two or more score 0.0. The first error text is quoted in the report detail.
j_empty_stateWhat a fresh install looks like before any sync: an honest empty state rather than phantom data or a blank page.The scorer boots a second instance of the app against a brand-new empty database on its own port, waits for /api/health, then probes it in headless Chromium, polling up to 8 s for either rendered rows or visible empty-state text matching 'no payments / nothing / empty / no records|results|data|items|transactions'.Zero rendered data rows plus visible empty-state text; when the probe exercised the empty-state call-to-action (emitted emptyCtaWorks), the Sync CTA must actually work.Any rendered row on the empty database scores 0 (phantom data). With the CTA exercised: empty text but a dead CTA scores 0.6. A page that renders content (bodyTextLength > 0) but has no matched empty-state text scores 0.3; a blank page scores 0.0. If the empty instance dies at boot or never becomes healthy the check refuses as PROBE UNAVAILABLE.
j_error_stateWhat a user sees when the backend is unreachable: a visible, actionable error state instead of a blank or silently broken page.The probe loads the page in headless Chromium with --block-api: the document itself is served, but every /api, /data, /graphql and .json request is aborted at the browser network layer (connectionrefused), so the app's own error UI is what gets measured; banner detection requires error phrasing and explicitly ignores bare status vocabulary like a 'Failed' filter chip.A visible error element whose text contains actionable phrasing (try again / retry / check / refresh / running); when the probe also exercised retry (emitted retryRecovered), the retry must actually recover.Ladder: 0.6 for visible + actionable where an exercised retry did not recover; 0.3 for any visible error indication without actionable text; 0.0 for no error state at all (backend failure leaves a blank page). The retry rung is authoritative only when the probe emitted the field, so an unmeasured retry never blocks 1.0.
j_first_useThe first-visit experience as one property: the page shows real data quickly, the on-page total agrees with the fixture, and the console stays clean.Playwright headless Chromium loads the app at 1280x800 with an init-script MutationObserver that stamps performance.now() at the first visible data row (rows must have client rects and visibility, so hidden fallback tables never count), then harvests the DOM-claimed record total and console/page errors.Gate: at least one visibly rendered data row. Then min() of three components must be 1.0: time-to-first-data <= 2000 ms, the DOM-claimed total equals the fixture's 1553, and zero console errors during the load scenario.Compound gate x min: zero rendered rows scores 0 outright; otherwise the weakest component bounds the whole. The first-data ladder steps 1.0/0.75/0.5/0.25 at 2000/3000/4500/8000 ms (0 beyond or never); a wrong or missing claimed total sets reconciliation to 0; any console error sets console_clean to 0. Partial credit never stacks across components.
j_loads_datadiagWhether the app renders any payment rows at all, and whether the count it claims on the page matches the fixture truth.The same headless-Chromium load probe counts visibly rendered table rows (client rects plus computed visibility) and extracts the claimed total from visible non-table text via patterns like 'of N' and 'N payments', after scrubbing dates and currency amounts so a money figure cannot masquerade as a row count.At least one visibly rendered data row (0.5) plus a DOM-claimed total exactly equal to the fixture's 1553 records (0.5).Each half is independent: no visible rows loses 0.5; a claimed total that is absent or differs from 1553 loses the other 0.5. Rows present only in a hidden DOM subtree count as zero.
j_sync_journeyThe spec's headline interactive flow: a user clicks Sync Now and the app visibly fetches, indicates progress, finishes, and refreshes the view.The app is booted against a half-capped vendor, the cap is lifted, then the probe finds and clicks #sync-now (or any visible /sync/i-labelled button) in headless Chromium and requires causal evidence of completion: an observed /api/sync entry in the browser's resource timing, an in-flight disabled state, or a changed view snapshot (row count, last-sync text, or a hash of visible table content).All four quarter-weighted parts true: button found and clickable; an in-flight state (disabled/aria-busy/data-state=syncing) within 1200 ms of the click; completion (button re-enabled with no error banner, corroborated by at least one piece of causal evidence -- an uncorroborated 'completed' is demoted to false); and a refreshed view (row count, last-sync text, or visible table hash changed).Each missing part costs 0.25: no findable sync button scores 0; no in-flight state, an uncorroborated or failed completion, and an unchanged view each cost their quarter. The check refuses (PROBE UNAVAILABLE, excluded from the tier mean) when the half-seed vendor surface is missing, because view refresh would be unobservable.

VRendered truthweight 0.08 · 7 checks

CheckWhat it measures · howEarns 1.0Loses points
v_currency_renderedWhether rendered money is arithmetically correct per currency -- right digits and right decimal exponent -- with the zero-decimal JPY and three-decimal KWD traps weighted separately.The probe harvests raw amount-cell texts from up to 12 visibly rendered rows (cells carrying a currency token or symbol plus digits); the scorer detects each cell's currency (EUR/USD/JPY/KWD incl. symbols) and pairs it against the fixture's first-page amount_minor values -- digit-for-digit equality with the minor amount and decimal places equal to the currency exponent, never a string match on the spec's examples.Score = 0.6 x fraction of cells exponent-correct + 0.4 x trap score; full marks need every graded cell to match a fixture minor amount with the correct exponent AND both traps clean: all JPY cells with zero decimals (a trailing 3-digit group reads as thousands grouping and passes; 1-2 trailing decimals is wrong money) and all KWD cells with three decimals, with both currencies actually present.No amount cells rendered scores 0. Each wrong cell lowers the 0.6 fraction; each trap currency that is absent from the rendered page or has any wrong cell zeroes its half of the 0.4 trap component. Refuses as unavailable if the fixture amount ground truth is missing.
v_dates_readableWhether the Date column shows humans a readable date instead of a raw machine timestamp.The load probe locates the date column in the first visibly rendered row and captures the innerText of the first three rows' date cells from real headless-Chromium computed rendering; the scorer regex-classifies those strings.At least one date cell rendered, no cell containing raw ISO-8601 (YYYY-MM-DDTHH:MM), and at least one text that is recognizably human (month letters, or a d/m or d.m numeric pattern).No date cells rendered scores 0; any raw ISO-8601 timestamp shown to users scores 0 (the spec forbids machine timestamps in the rendered page); formatted-but-not-recognizably-locale-readable text scores 0.5.
v_filterWhether the status filter drives the product end to end: rendered rows, the total readout, the control's selected value, and a clean restore to All.The probe drives the #status-filter custom dropdown the way a user would in headless Chromium -- opens the control, clicks the 'refunded' option (found via data-option/data-value/[role=option] inside the control, or exact visible text anywhere so portaled dropdowns work), snapshots rendered row statuses and the 'showing X-Y of TOTAL' readout, then restores All -- and the scorer judges the raw snapshots against the fixture's per-status totals.All four contract parts: every rendered row shows 'refunded' with at least one row; the readout total equals the fixture's expected refunded count; the control's data-value attribute reads 'refunded' after the pick; and restore returns data-value to empty with the readout back at 1553.Rows follow the filter but the readout/data-value/restore contract fails: 0.5. Control present but the pick changed nothing, or its options were not drivable by the probe: 0.2 (a dropdown a probe cannot open is a dropdown many users cannot open). No status filter control: 0.0. A control that exists but was never exercised by the probe refuses as PROBE UNAVAILABLE rather than guessing.
v_paginationWhether the list is actually paginated in the rendered page: visible controls plus a bounded first view instead of an unpaginated dump.The load probe searches the rendered DOM for visible prev/next/first/last/page-N buttons or aria-labels, a nav[aria-label*=pag] / [class*=pagin] container, or a 'showing X-Y of N' readout, and counts visibly rendered rows in the same headless-Chromium session.Pagination controls visible (0.6) plus a rendered row count that is greater than 0 and at most the spec's default page size of 50 (0.4).Missing controls loses 0.6; zero rows or more than 50 rows in one view loses 0.4 -- the finance team scrolling an unpaginated dump earns neither part.
v_responsive_375Whether the page survives a phone-width viewport: no horizontal scrolling and real content still rendered at 375 px.The probe resizes the headless-Chromium viewport to 375x812, reloads the page, and compares document scrollWidth against window.innerWidth (+1 px tolerance) while re-counting visibly rendered rows.No horizontal scroll and at least one rendered row at 375 px; when the probe measures tap targets (emits tapTargetsOk) they must also be adequately sized -- the current probe v2 does not emit that field, so full credit stands on no-scroll plus rows, with the report noting tap targets were not measured.Any horizontal scroll at 375 px scores 0.0 (the page breaks on a phone). No scroll but zero rendered rows also scores 0.0 -- empty pages never scroll, so the pass would be vacuous. Small tap targets, when measured, cap the score at 0.6.
v_status_distinctWhether the four payment statuses are visually distinguishable at a glance and painted in the spec's frozen palette.The probe reads getComputedStyle color and background of the innermost element carrying each status label (walking up through transparent backgrounds) for every distinct status on page one; the scorer compares against the frozen hexes -- settled rgb(22,163,74), pending rgb(245,158,11), refunded rgb(139,92,246), failed rgb(220,38,38) -- with a tolerance of 8 per RGB channel on either the background or the text color.Score = 0.6 x (statuses at their frozen hex / max(3, statuses rendered)) + 0.4 x distinctness; full marks need at least 3 statuses rendered (the v3 fixture interleaves statuses so the default first page always carries at least 3), every one at its frozen hex, and every (color, background) pair unique.No status cells rendered scores 0. Each status off its frozen hex lowers the 0.6 component; fewer than 3 rendered statuses or any two statuses sharing an identical style pair zeroes the 0.4 distinctness component.
v_stylingWhether the page is deliberately styled rather than browser-default: a real stylesheet, layered surface colors, a chosen font, and a branded header.The probe inspects the rendered page in headless Chromium for a non-empty style block or linked stylesheet, counts distinct computed non-transparent background colors across header/table/button/card-class elements, reads the body's computed font-family, and checks for a visible #app-header (falling back to a source grep of the shipped HTML when the probe did not emit the field).All four weighted parts: stylesheet present (0.3), at least 3 distinct background colors (0.25), a body font that is set and not Times (0.25), and a branded #app-header (0.2).Each absent part loses its weight independently: no stylesheet -0.3, fewer than 3 distinct backgrounds -0.25, empty or Times-default font-family -0.25, no visible app header -0.2.

PPerformanceweight 0.06 · 6 checks

CheckWhat it measures · howEarns 1.0Loses points
p_buckets_latencyThe 3D panel's data source is fast: p95 latency of GET /api/buckets against a 200 ms budget.Same instrument as p_list_latency: 20 sequential harness-side HTTP GETs of /api/buckets, errors excluded, p95 over successes; fewer than 10 successes makes the percentile None and the score 0.p95 <= 200 ms (top rung tightened to 200/k_P when calibrated).<=400 ms earns 0.75, <=800 ms 0.5, <=1600 ms 0.25, slower or unmeasurable 0. Below 1.0 also locks this check's share of the E gate.
p_first_frameTime from navigation to the first real 3D draw call on the WebGL canvas, against a 3000 ms budget.Headless Chromium viz scenario with deterministic SwiftShader GL: an init script wraps HTMLCanvasElement.getContext before any app code and shims drawArrays/drawElements (+Instanced variants) to stamp performance.now() per draw call; firstDrawMs is the first stamp on the page's own clock — clears never count.First instrumented draw call at <= 3000 ms (3000/k_P when calibrated).<=5000 ms earns 0.75, <=8000 ms 0.5, <=15000 ms 0.25. A canvas that never draws scores 0 (app truth). A viz probe error, a probe build that emits no firstDrawMs field, or the stamp lost to the probe's 150 s hard cap is PROBE UNAVAILABLE — the row is excluded from the tier mean, never converted to a zero. Below 1.0 also locks this check's share of the E gate.
p_list_latencyThe payments list endpoint answers a default page quickly: p95 latency of GET /api/payments?limit=50 against the spec's own 150 ms budget.20 sequential HTTP GETs from the harness process (Python urllib, idle phase after all sync work); failed requests are excluded from the distribution, and if fewer than 10 succeed the p95 is reported as None and scores 0. p95 is the sorted-index method over the successes.p95 <= 150 ms (top rung is 150/k_P once thresholds are calibrated; k_P=1.0 in the current rc defaults). A 1.0 here also unlocks this check's share of the proportional excellence gate.Quantized quarters, no interpolation: <=300 ms earns 0.75, <=600 ms 0.5, <=1200 ms 0.25, slower (or p95 unmeasurable because most requests failed) 0. Anything under 1.0 also locks this check's slice of the 0.12 E-tier weight.
p_page_interactiveTime from navigation to the first VISIBLE data row the user could actually see, against a 2000 ms budget.Headless Chromium (Playwright) load scenario: a MutationObserver installed before any app script stamps performance.now() the instant the first visible table row (>= 2 cells, non-empty text, real client rects, not visibility:hidden) lands in the DOM — rendered-means-seen, so rows inside a hidden fallback table never count and probe polling latency never enters the number.First visible data row stamped at <= 2000 ms after navigation start (2000/k_P when calibrated).<=3000 ms earns 0.75, <=4500 ms 0.5, <=8000 ms 0.25. No visible data row within the poll window — or a failed load probe — scores 0 (this row does not refuse on probe error; the value is simply absent). Below 1.0 also locks this check's share of the E gate.
p_summary_latencyThe summary endpoint answers quickly: p95 latency of GET /api/summary against a 150 ms budget.20 sequential harness-side HTTP GETs of /api/summary, errors excluded from the percentile, p95 over successes; under 10 successes reports None and scores 0.p95 <= 150 ms (150/k_P at the top rung when calibrated).<=300 ms earns 0.75, <=600 ms 0.5, <=1200 ms 0.25, slower or unmeasurable 0. Below 1.0 also locks this check's share of the E gate.
p_sync_wallWall-clock time for one full POST /api/sync, against a 90 s budget that includes the vendor mock's documented waits (Retry-After, stalls).One harness-side POST /api/sync, wall-timed with a 240 s timeout, in the idle perf phase after the graded sync runs.Sync completes in <= 90000 ms (90000/k_P when calibrated).<=135000 ms earns 0.75, <=180000 ms 0.5, <=270000 ms 0.25, slower 0. Below 1.0 also locks this check's share of the E gate.

T3D panelweight 0.14 · 7 checks

CheckWhat it measures · howEarns 1.0Loses points
t_cameraThe orbit camera implements the documented controls: wheel zoom, drag-to-rotate at 0.35 deg/px, and double-click reset.Real synthetic input in headless Chromium, verified by framebuffer readback against re-projected expectations: mouse.wheel(0,400) must land the scene at dist 30·exp(0.0012·400) ≈ 48.5; a scripted drag of 60 delivered 2 px moves (~2 s, +120 px) must land yaw at 35 − 120·0.35 = −7°; a mid-drag framebuffer sample proves the scene actually moved; double-click must restore the baseline projection. The canvas rect is re-measured after the wheel so a page that scrolls instead of zooming is caught.Wheel reprojection fraction 1.0 (0.20) + drag reprojection at the expected yaw 1.0 (0.35) + vsdbg camera yaw within 1.5° angular distance of −7° (0.10) + reset restoring baseline 1.0 (0.35).Each fraction scales its weight. A sign-flipped drag (yaw +42° instead of −7°) earns only half credit on the drag term. Reset credit pays ONLY if the scene provably moved during the drag (mid-drag sample or projection fractions) — a static canvas cannot cash the reset anchor. A canvas that cannot be fully scrolled into the viewport leaves the interaction sections absent and scores 0 (app truth). Probe error or timeout-lost sections are PROBE UNAVAILABLE.
t_context_realThe #viz3d panel is a real, drawing WebGL surface — not a styled div, an image, or an unused canvas.Headless Chromium with wrapped getContext plus a WebGL readPixels readback of the whole framebuffer: the probe verifies a webgl/webgl2 context was created on the canvas with id viz3d, compares the backing-store size to the CSS rect x devicePixelRatio, counts instrumented draw calls, samples a blind 6x4 grid of pixels, and reads the four 3 px-inset corners.All five parts: a WebGL context on #viz3d (0.25), backing store matching rect x DPR within 1 px (0.15), >= 1 real draw call (0.15), grid coverage with >= 3 non-background samples in >= 2 distinct colors (0.25), and all four corners reading the spec background #0F172A within tolerance 8 (0.20).Weighted sum — each failed part drops its weight (e.g. a context on the wrong canvas loses 0.25; a wrong-size backing store 0.15; a canvas painted edge-to-edge in one color fails both coverage and corners). Probe error or the section lost to the viz hard cap is PROBE UNAVAILABLE, excluded from the tier mean.
t_fallbackThe 2D escape hatch works both ways: a user can toggle to a data-correct 2D table, and a machine without WebGL degrades gracefully instead of crashing or going blank.Two browser runs. Toggle half: in the normal viz run the probe clicks #viz-toggle and checks every #viz-fallback cell (keyed data-day|data-status) against the expected non-zero bucket counts. Grace half: a separate headless run whose init script makes getContext return null for any WebGL type (canvas-2D stays alive), then grades console errors, the auto-shown fallback table, the payments table, and an explanatory notice — credited only when visible near the viz panel under glKill AND absent in the normal run (the differential kills static-footer freeloading).Toggle half (0.5): all fallback cells correct. Grace half (0.5): zero console errors (0.4 of it) + all auto-fallback cells correct (0.35) + the differential notice (0.15) + payments rows still rendering (0.10).Each wrong cell shrinks its half's fraction; a visible toggle whose table never shows earns only 0.1 on that half. If the no-WebGL scenario is unavailable the check pays at most 0.5 x toggle (the unproven half licenses nothing) or refuses entirely when the toggle is also unmeasured. Probe error on the main viz run is PROBE UNAVAILABLE for the whole check.
t_pickingClicking a 3D bar drives the actual product: the status filter takes the bar's status and the payments table really refreshes to match.Real mouse clicks at analytically computed bar-top screen coordinates (up to 5 strided targets) in headless Chromium. A pick counts only if #status-filter's data-value attribute becomes the bar's status (data-value ONLY — no aria/text freeload) AND, when the probe measured it, the visible rows are uniformly that status and the 'showing X–Y of TOTAL' readout equals the fixture's per-status total. A depth pair clicks a pixel where a nearer bar of a different status occludes another bar's top.Every pick correct (0.65) and the depth-pair click selecting the OCCLUDING bar with the same table proof (0.35); when no cross-status occlusion exists on screen, the depth term falls back to the pick fraction.Score is 0.65 x correct-pick fraction + 0.35 x depth term. A filter that takes the value but leaves the table stale fails that pick; a depth-unaware ray pick (selecting the farther bar) zeroes the 0.35. Probe error or a timed-out picking section is PROBE UNAVAILABLE.
t_scene_bindingThe rendered 3D scene actually encodes the bucket data: every bar in the right cell, at the right height, in the right status color, with sky where nothing should be.Analytic pixel recomputation: the probe rebuilds the expected scene node-side from the fixture's BENCH_VIZ_BUCKETS with the spec camera (yaw 35, pitch 27, dist 30), projects bar-top / above-top / side-face / sky sample points, drops occluded, backfacing, sub-25 px² and edge-ambiguous samples via ray-cast clearance, then does one WebGL readPixels and judges each sample by 3x3 pixel majority within color tolerance 8.All graded sample classes match: tops weighted 0.50, above-top background 0.25, side faces 0.15 (normalized by the 0.75 achievable fraction for antialiased edges), sky 0.10 — weights renormalize over classes that have samples, so a legal layout with no sky samples is not penalized.Each missed sample lowers its class fraction. Fewer than 3 distinct top colors when >= 3 statuses exist caps the score at 0.25 (mono-wash guard). An empty-database run where the vsdbg API still claims bars scores 0 outright (phantom data). Zero graded samples scores 0. Probe error or a timed-out section is PROBE UNAVAILABLE. Also an E-gate condition: only a full 1.0 unlocks its share of the excellence slice.
t_tooltipHovering a bar shows a truthful tooltip fast, and it goes away when the pointer leaves.A real mouse move to the largest-area bar top in headless Chromium; #viz-tooltip visibility is polled, its latency taken from page-side stamps (a pointermove listener's performance.now() to the MutationObserver stamp of the mutation that made the tooltip visible — probe RPC latency never enters the rung), then the mouse moves off-canvas and the tooltip must hide.Tooltip text starts with '<count> <status>' for the hovered bar (0.5), appears within 150 ms by page-side stamp (0.3 at full; calibration-owned rungs), and hides off-canvas (0.2).Never appearing scores 0 outright. Wrong leading text loses the 0.5. Latency 150–400 ms earns 0.6 of the 0.3 term, 400–1200 ms 0.3 of it, slower 0. Not hiding loses the 0.2. Probe error or a timed-out tooltip section is PROBE UNAVAILABLE.
t_vsdbg_truthThe mandated window.vsdbg instrumentation API tells the truth about the scene the canvas actually shows — scene graph, projection math, and picking all agree with independently recomputed geometry.In-page evaluation calling the app's own vsdbg.scene()/project()/pick()/camera() and comparing every answer against the probe's node-side analytic pipeline: scene bars matched on key + count + exact x/z/h, projections checked at up to 6 spot bars against the probe's own projectPt, picks checked at up to 4 top centers.version == 3 (0.10), every expected bar matched with no overclaim (0.40 — the denominator is max(expected, claimed), so inventing bars costs), max projection error <= 2.0 px across all spots (0.30, all-or-nothing), and every vsdbg.pick returning the expected bar key (0.20).Weighted sum: a missing or wrong-version vsdbg drops 0.10; each unmatched or overclaimed bar shrinks the 0.40 scene term; projection error over 2 px zeroes the 0.30; each wrong pick shrinks the 0.20. window.vsdbg absent scores 0 (app truth). Probe error or timeout-lost section is PROBE UNAVAILABLE.

HARDHard mechanismsweight 0.20 · 5 checks

CheckWhat it measures · howEarns 1.0Loses points
h_conflict_danceOptimistic concurrency done right: every vendor write carries If-Match, a 412 is recovered by re-reading and retrying exactly once, and a persistent conflict surfaces to the app's caller as 409 with the row untouched — done wrong, it silently loses someone's edit.The harness arms a one-shot injected 412 on the vendor mock, POSTs a note to the app's /api/payments/{id}/note, then arms always-contended mode and posts a second note. Evidence comes from the vendor's PATCH log via conflict_trace() — 428 responses count writes missing If-Match, and the recovery is validated by matching the retry's If-Match header against the fresh version number the 412 response disclosed — plus the app's local HTTP status on the second write and a follow-up GET proving the row's note is unchanged.All four weighted parts: If-Match on every vendor write (0.25), the injected 412 recovered by exactly one retry carrying the fresh version and succeeding with 200 (0.35), no retries beyond that one (0.15), and the always-contended second write returned local 409 with the row's note unchanged (0.25).Weighted sum, so each miss costs its weight: any bare write without If-Match loses 0.25; failing to recover the 412 — or retrying blind without the fresh version — loses 0.35; a retry storm loses 0.15; mapping the persistent conflict to a 500, a fake success, or a corrupted row loses 0.25. A missing conflict trace (the 412 surface never exercised) is PROBE UNAVAILABLE.
h_durabilityThe compound durability contract: data survives a hard restart, survives two operators syncing at once, and the store's write path is genuinely atomic.Three instruments joined by gate × min: the SIGKILL+reboot probe's surviving row count measured against EXPECTED_TOTAL = 1553; two overlapping POST /api/sync threads fired simultaneously, then /api/health's total; and static analysis of store.py — a regex over the INSERT INTO payments statement distinguishing ON CONFLICT ... DO UPDATE from INSERT OR REPLACE/IGNORE.The had_rows_before_kill gate holds and min(persists, concurrent, atomic) = 1.0: all 1553 rows present after kill+reboot, exactly 1553 rows after the two concurrent syncs, and a true ON CONFLICT DO UPDATE upsert in the store.min() of the three components. persists is the surviving-row fraction; concurrent decays linearly with |total − 1553| / 1553, so duplicated or lost rows under the race cost proportionally; atomic is 0.5 for INSERT OR REPLACE / OR IGNORE (writes but does not merge) and 0.0 with no upsert at all. No rows before the kill fails the gate and scores 0. Missing fixtures are PROBE UNAVAILABLE.
h_sync_disciplineThe compound sync contract: a second sync must be cheap (conditional requests and 304s), idempotent (inserts nothing, total unchanged), and vendor-side updates must propagate — together, not severally.Three instruments joined by gate × min: the vendor mock's JSONL trace, sliced by phase markers around the graded second sync, counts its list requests, how many carried If-None-Match, and how many were answered 304 (the window is explicitly closed after sync #2 so later phases cannot dilute the evidence); the app's sync #2 JSON body supplies inserted and total against EXPECTED_TOTAL = 1553; the propagates component reuses the mutate_statuses pass — per-id local GETs after sync #3.Both gates hold (sync #2 ran and the vendor saw requests during it) and min(cheap, idempotent, propagates) = 1.0, where cheap = 0.5·(conditional/requests) + 0.5·(304s/requests) — every sync-#2 request conditional and 304'd — idempotent requires inserted == 0 with total exactly 1553, and propagates requires 12/12 mutated statuses visible locally.min() means the weakest leg is the score: an unconditional full re-walk scores cheap ≈ 0 and caps the whole check near 0 even with perfect idempotence; total correct but inserted ≠ 0 makes idempotent 0.5; unpropagated mutations cut proportionally. A failed gate (sync #2 never ran, or no vendor requests observed in its window) scores 0. Missing fixture constants are PROBE UNAVAILABLE.
h_webhook_ledgerThe app correctly consumed untrusted push traffic: an exact received/applied/ignored/rejected ledger, a forged delivery rejected, and a stale delivery ignored without touching the rows.After the restart phase (so registration must survive a reboot), the vendor mock executes its fixed 8-step HMAC-signed delivery script against the app's registered webhook URL: 4 genuine applies, 1 stale snapshot delivered late, 1 byte-identical duplicate redelivery, and 1 forged event signed with the wrong secret whose claimed mutation never happened. The scorer then reads the webhook counter quad from /api/health, takes the forged delivery's HTTP status straight from the script results, and verifies each touched payment's version and note over the app's own API against the fixture's simulated end state.The registered gate holds (health reports a registered webhook) and min(counters, forged_rejected, stale_ignored) = 1.0: counters exactly {received: 7, applied: 4, ignored: 2, rejected: 1}, the forged delivery answered with HTTP 401, and every touched row's version+note matches the simulated final state — proof the stale and duplicate events were ignored on the app's actual rows, never a proxy.The counters component is the fraction of the four counters that are exact (0.25 each), but min() rules: a forged delivery not answered 401 zeroes the check, as does any row betraying an applied stale event. An unregistered webhook fails the gate and scores 0. The script never running, or the expected-counters fixture missing, is PROBE UNAVAILABLE.
request_efficiencyThe first sync fetched the complete collection in the optimal number of vendor requests while passing every vendor trap — fetching nothing is not efficiency, so credit requires the data actually arrived.Counts sync #1's GET list requests inside its phase-marked window of the vendor mock's JSONL trace and compares against the mock's own simulated perfect-client walk — OPTIMAL_REQUESTS = 20 for the full 1553-row collection, OPTIMAL_REQUESTS_HALFSET = 13 for the half-capped 776-row collection sync #1 actually runs against (both computed from page size plus trap overhead, never hand-written). Completeness is the synced total divided by what the vendor was serving at the time; traps_ok requires all three trap-ledger fractions at 1.0.Requests exactly equal the applicable optimum AND completeness = 1.0 AND every vendor trap passed. Fewer-than-optimum requests earn 1.0 only under the same complete-and-all-traps condition — otherwise under-optimum scores 0, because undercutting the optimum means data was skipped.Rungs multiply by completeness: exact optimum without all traps passed earns 0.5·completeness; optimum+1..2 earns 0.75·completeness; optimum+3..7 earns 0.5·completeness; beyond that the score decays as (optimum/requests)·completeness. Any failed trap caps the entire check at 0.5 regardless of branch (amendment F15b: no branch awards ≥ 0.75 with a failed trap). Zero observed vendor requests score 0; missing fixture constants are PROBE UNAVAILABLE.

EExcellenceweight 0.12 · 4 checks

CheckWhat it measures · howEarns 1.0Loses points
e_frames_under_dragThe 3D view stays interactive under input: real frames rendered during the scripted drag, in the spec's own unit (frames per drag), with proof the frames actually drew something.vsdbg.frames() sampled before and after the scripted 60-move ~2 s drag in headless Chromium, cross-checked against the glInstrument draw-call delta — a frame counter that ticks with zero instrumented draw calls (an empty rAF loop) earns nothing.>= 24 frames over the drag with a positive draw-call delta (calibration-owned rungs, frozen at the Bedrock fit).>= 18 frames earns 0.75, >= 12 earns 0.5, >= 6 earns 0.25, fewer 0. Unmeasurable frames or zero draw calls scores 0. Probe error or the drag section lost to the hard cap is PROBE UNAVAILABLE, excluded from the E mean.
e_hard_masteryExcellence includes the mechanisms, not only the surface: mastery of the whole HARD tier (sync discipline, durability, webhook ledger, conflict dance, request efficiency).Computed from the scorer's own already-graded HARD rows in the same run — the mean of every HARD check that was measurable (unavailable rows excluded).HARD-tier mean >= 0.90.Below the 0.90 cliff the score is (mean/0.90) x 0.5 — a 0.89 HARD mean earns ~0.49, so near-mastery pays at most half. No measurable HARD rows at all is PROBE UNAVAILABLE. Like all E rows, the payout is further scaled by the proportional excellence gate (journey checks at 1.0, zero nominal console errors, v_responsive_375 / v_dates_readable / t_scene_binding at 1.0, and every P check at 1.0 each unlock their share of the 0.12 slice).
e_optimistic_paintEditing a payment note paints optimistically: the new value appears in the cell while the write is provably still on the wire, then really saves.Causal proof, not a race: the probe's page.route holds the POST /api/payments/{id}/note response for 800 ms, clicks the Note cell, types a value, presses Enter, and checks the cell at +250 ms while the route is verifiably still pending. Paint latency comes from a page-side MutationObserver stamp; a pre-existing draft value (contenteditable caveat) earns no latency credit.Painted-while-held (the causal gate, 0.5 base) + a visible saving/pending indicator during the hold (0.15) + paint stamp <= 100 ms (0.15, calibration-owned rungs) + the value confirmed saved after the hold releases (0.2).No Note cell, a dead editor affordance, or a cell that waits for the network each scores 0 outright. Paint at 100–250 ms earns 0.6 of the latency term, 250–800 ms 0.3, slower or pre-existing 0; missing saving-state or failed save drops those terms. Probe error is PROBE UNAVAILABLE.
e_under_load_latencyThe API stays fast AND correct while a sync is actually in flight — read p95 under concurrent load, with the load itself proven.Harness-side concurrency: 8 Python reader threads issue 6 GET /api/payments?limit=50 each (48 samples) while POST /api/sync runs in another thread; the vendor mock's armed stall window guarantees the sync is long enough to overlap, and every sample is timestamped so the overlap fraction is a measured fact. Correctness requires each response to be a 200 with a well-formed body and exactly 50 rows.Overlap fraction >= 0.5, every response correct, and p95 <= 150 ms (calibration-owned rungs, top tightened by k_P).p95 <= 300 ms earns 0.75, <= 600 ms 0.5, <= 1200 ms 0.25. ANY wrong/empty response under load scores 0 — fast wrong answers are not performance. If fewer than half the reads provably overlapped the sync, the check REFUSES (PROBE UNAVAILABLE) rather than crediting or zeroing on unproven load.

§10 · sb-7 · The task

sb-7 — the task every entrant receives

Build `app` — the Meridian Payments Console: two cooperating Python-stdlib services (ledgerd + notifierd) that sync 12,288 payments from the mock Meridian API v3, keep them consistent through vendor-pushed webhooks, concurrent edits, SIGKILLs and partitions, run a maker/checker approval workflow that creates real vendor payments, and give a finance team a live console — a payments table, a notifications feed, and an interactive raw-WebGL 3D field rendering every payment as an instanced column.

The specification below is the prompt: every entrant — a cloud model in a single session or the local fleet — receives the same frozen document with the mock vendor's documentation URL, base URL and API key rendered in. Nothing else is provided. (The frozen spec is roughly 8,200 words (~54 KB, 873 lines of markdown) — the entrant also has to read the separately rendered Meridian API v3 docs at {DOCS_URL}, which the spec makes mandatory before starting.)

DeliverableRole
python -m app / app.ledgerd / app.notifierdThe boot contract: three commands the harness drives independently — the combined wrapper, ledgerd alone (--db-dir, --port, --notifier, --vendor, --tokens-file) and notifierd alone. Both services bind 127.0.0.1 and are listening within 10 seconds, boot cleanly with the vendor down AND with the other service down, and resume cleanly against an existing --db-dir. On boot ledgerd starts its first sync unprompted, retrying at least every 5 s while the vendor is unreachable.
ledgerd · ledger.dbVendor sync (the 192-page walk, reversals, conditional resyncs under the collection-generation rule), the append-only event ledger (seq contiguous from 1 — a gap is a lost write), the transactional outbox with a background relay (batches ≤ 50, backoff ≤ 2 s, at-least-once), the JSON API (payments, summary, buckets, drafts, events, outbox status, notifications proxy, viz records, SSE stream), the signed-webhook endpoint, and the static frontend. Owns ledger.db and never opens notifier.db.
notifierd · notifier.dbThe idempotent consumer: POST /notify/events dedupes on ledger seq against a DURABLE processed set (exactly-once is graded on it across SIGKILLs); exactly four event types materialize exactly one notification row each (draft.submitted, draft.approved, draft.rejected, reversal.created — payment.sent is processed but produces none). Owns notifier.db.
app workflow — maker / checker / adminThe approval workflow over three bearer tokens: draft → submitted → approved|rejected (approved → sent once the vendor accepts). Four-eyes: the submitter's own token cannot approve or reject (403 approval_forbidden). On approve the app creates the REAL vendor payment with a stored Idempotency-Key — a crash mid-send retries with the SAME key; exactly one vendor payment per approved draft, ever.
web/index.html · styles.css · app.js · viz.jsThe console page, four files, ≤ 150 KB combined, zero external code: branded #app-header, per-currency summary (never a cross-currency figure), the 3D field panel, a server-driven payments table (pagination, sortable Date/Amount headers with aria-sort, custom status/currency dropdowns exposing data-value, optimistic inline note editing), the notifications feed (poll ≤ 5 s, data-state live/degraded), and the drafts panel driving the full approval journey through the UI alone.
web/viz.js — the 3D fieldRaw WebGL, no libraries: every one of the 12,288 payments as one instanced column on a day × in-day-rank grid, ≤ 8 draw calls per frame, demand rendering (0 draws at rest), a GPU pick buffer (identity colors, depth-correct), an inertial orbit camera with a printed coast law (τ = 0.4 s), 12 collision-culled DOM labels, a brush linked both ways to the table, SSE streaming diffs with byte-accounted uploads, and the graded window.vs7dbg instrumentation.
DECISIONS.mdThree judgment corners the spec deliberately leaves open, each decided, shipped and documented under headings ## D1 (does the brush survive a streamed mutation of a brushed record?), ## D2 (is a rejected draft terminal or resubmittable?), ## D3 (does the table render empty-with-progress or block before the first sync?). Either answer passes; an undocumented corner, or a document contradicting observed behavior, does not — the run exercises all three.
README.mdThe exact commands to install nothing, run both services (together and separately), and sync.

The vendor

The app must consume Meridian API v3 exactly as its rendered docs prescribe: fixed-size pages of 64 (no limit parameter — 12,288/64 = 192 pages), resume after a dropped connection, honour Retry-After on a 500 (one documented retry, never a fresh unconditional restart of committed work), restart the cursor on 410 cursor_expired, and make later syncs cheap with ETag/If-None-Match PLUS the X-Collection-Generation rule: a 304 whose generation disagrees with the stored one is a cache miss — drop the validator and refetch unconditionally, exactly once; more than 3 identical conditional requests in a row is the infinite-loop bug, and serving stale data as fresh after a mismatched 304 is worse. The walk is NOT snapshot-isolated: the vendor commits mutations and creates mid-walk and delivers their webhooks before, between and after the page bodies they race — upserts compare version, never blind-write. Every vendor request times out at ≤ 10 s. Writes are optimistic-concurrency only (If-Match, one retry on 412, a second 412 surfaces as 409 conflict; a write without If-Match earns the vendor's 428 — a bug in your client). Approved drafts create real vendor payments under a stored Idempotency-Key that every retry MUST reuse — a fresh key per retry is the seeded duplicate-payment bug.

Webhooks and the approval workflow

Webhooks reverse the arrow — the vendor calls YOU: after binding, the app registers its endpoint (idempotent by URL, retried until the vendor is reachable), answers the unsigned webhook.verify challenge by echoing the hex, then treats every delivery as untrusted until Meridian-Signature (t=<unix>,v1=HMAC-SHA256 of "<t>.<raw body>" over the raw bytes) verifies. Valid events apply idempotently and in version order — the vendor WILL deliver duplicates, WILL deliver v+2 before v+1, WILL (once) forge a signature, and WILL race the sync walk; the four live health counters (received/applied/ignored/rejected) are the ledger of how the app handled all of it. Refunds arrive as a two-part transaction group (status flip + reversal, same txn id, parts in either order, duplicated like any delivery) that MUST apply atomically — no API read may ever observe half a group: a summary showing the refunded payment without its reversal total is the graded failure.

The 3D field

The 3D field renders EVERY payment as one instanced column on a day × in-day-rank grid — raw WebGL on the main thread ({antialias: false, alpha: false}), no three.js, no library; the 150 KB asset budget enforces it. Every contract is frozen and independently recomputed by the grader: cell pitch Δ = 1.2, footprint 0.9 × 0.9, height h = clamp(0.9 + 0.55·log10(a_major), 0.2, 4.2) THROUGH each currency's minor-unit exponent (a client that forgets JPY = 0 or KWD = 3 renders measurably wrong heights, verified ±3 px on rendered column tops); flat unlit colors at the frozen status hexes with sides at 0.55× and a 0.30× brushed-dim; background #101828 with nothing else drawn. Instance identity is the stable arrival index n — never re-sorted. Rendering is bounded: ≤ 8 default-framebuffer draw calls per frame at full count (the wrapper counts every GL entry point), 0 draws in any 500 ms window at rest, streamed diffs upload at most |changed|·stride + 4096 bytes with no realloc. Picking is GPU truth from an offscreen identity-color pick buffer, depth-correct under occlusion, refreshed in ≤ 4 offscreen draws with 0 visible-canvas draws. The orbit camera (yaw 30 / pitch 40 / distance 260 defaults) obeys a printed projection contract, drag/wheel laws, and an inertial coast v(t) = v0·e^(−t/τ) with τ = 0.4 s — graded via a mid-coast remaining-coast identity, a flick-reality check, a slow-release check and a settle budget. The 12 highest-amount records get 110 × 18 px DOM labels, occlusion-culled through YOUR pick buffer and collision-culled in priority order — never nudged. window.vs7dbg (layout, sceneDigest, camera, setCamera, pick, pickPixel, brush, frames) must tell the truth; an instrumentation layer reporting a scene the canvas does not show scores as broken, not as clever.

The data is hostile

The fixture is 12,288 payments spanning 96 consecutive Europe/Berlin calendar days containing exactly one Berlin DST transition — which one is seeded per run, so hardcoding it fails. The day a payment belongs to is the Berlin calendar date of its created_at INSTANT — not the raw string's date, not the UTC date; UTC-day bucketing produces measurably wrong counts AND measurably wrong 3D x-positions. Money is integers in minor units end to end with per-currency exponents (EUR/USD 2, JPY 0, KWD 3): a yen amount with two decimals or a dinar truncated to two is wrong money, there is no cross-currency total anywhere, and the 3D column heights go through the exponent on purpose. Values are seeded per run — build against the structure, never memorized constants.

The seeded fault schedule

A graded run executes a seeded fault schedule — positions change, boundaries do not: the vendor is DOWN for the first 3–8 s of boot (bind anyway, serve local data, sync unprompted when it returns); one dropped connection and one 500-with-Retry-After during the first walk; webhooks racing the walk plus one out-of-order pair, one duplicate, one forged signature, one mid-walk create and one two-part refund group; one 304 with a mismatched collection generation; ledgerd SIGKILLed mid-sync; notifierd SIGKILLed while ledgerd commits 8 more events (writes never block, pending grows, the feed shows degraded, the relay catches up in seq order after the heal); ledgerd SIGKILLed between an outbox commit and its delivery; and ledgerd SIGKILLed immediately after a draft submit's 200 and again after an approve's 200 with the vendor send still in flight — restart must finish the send with the SAME idempotency key. Convergence, not heroics at the moment of the kill, is what is graded.

Consistency, graded live

Ten consistency rules are graded continuously over the live run by replaying the app's event log and the notifier's processed set against the vendor's own commit ledger: no invented states (every applied (payment, version) exists in vendor history), per-key version order, monotonic reads, convergence at quiescence, transaction-group atomicity in EVERY summary snapshot, amount immutability, terminal conservation (per-currency counts and totals equal vendor ground truth, reversals included), no cross-currency sum anywhere, no lost acknowledgement (anything answered 2xx is present in final state), and exactly-once effects (no ledger event applied twice downstream, no doubled vendor payment, no doubled notification).

Boot contract

The harness boots the entrant's build with exactly the documented commands — `python -m app.ledgerd --db-dir P --port N --notifier … --vendor URL --tokens-file T` and `python -m app.notifierd --db-dir P --port M` — and kills/restarts the two services independently throughout the run. Both must be listening within 10 seconds of every start, must not crash when the database directory is fresh, when the vendor is unreachable, or when the other service is down, and every restart against the same --db-dir must resume cleanly: idempotent schema init, no data loss, no duplicate application of anything already committed. Nothing in a graded run is operator-driven — a run that needs a human to click, restart or nudge anything has already failed.

Performance budgets, as specified

  • Each service is listening within 10 seconds of process start.
  • First data rows render within 2 seconds of page load; the 3D field shows its first non-background frame within 3 seconds at full count.
  • GET /api/payments at limit=50 answers in under 150 ms at p95 — including while a sync runs with 8 concurrent readers and the SSE stream live; /api/buckets under 200 ms, /api/summary under 150 ms, /api/viz/records under 400 ms.
  • POST /api/sync completes the full 192-page walk within 120 seconds, documented waits included.
  • A streamed batch is applied — store, digest, pixels — within 250 ms of receipt; a camera change is visible within 250 ms; a scripted drag draws at least 0.8 frames per pointer move.
  • Draw accounting, always: at most 8 default-framebuffer draws per rendered frame; at most 8·(M+8) = 384 over the graded 40-move drag; 0 default-FBO draws in any 500 ms window at rest; a pick refresh costs at most 4 offscreen draws.
  • A coast settles within τ·ln(max(v0, 2)/2) + 0.7 s, capped at 2.5 s.
  • An optimistic edit paints within 100 ms of confirm — before the network responds.
  • The webhook endpoint answers within 3 seconds; the notifications feed polls at most every 5 seconds and recovers from degraded within 5 seconds of heal.
  • index.html + styles.css + app.js + viz.js total at most 150 KB uncompressed.

The spec demands the console be built as a product, not a debug view over the API: an intentional visual design with strong solid accent colors, a clear typographic hierarchy, and a branded header carrying the product name. Three prohibitions are explicit: never faded pastel washes (saturated solid colors over tints), never a left accent line or rail decorating cards or rows, and never browser-native controls where custom styling is expected — no default <select>, no alert()/confirm()/prompt(); filters are custom dropdowns exposing data-value and note editing is a custom inline editor with a non-blocking #notice (role="status"). Status badges use the four frozen hexes shared with the 3D field (settled #059669, pending #D97706, refunded #7C3AED, failed #B91C1C), distinct in computed color. Every user-visible timestamp renders human-readable in the user's locale — raw ISO-8601 must never appear. At a 375 px viewport the page lays out with no horizontal scroll and the canvas stays interactive at min height 240 px.

§11 · sb-7 · Composition

The sb-7 number, exactly

score = (0.88 × core + 0.12 × gate × excellence) × critical

score = 0.88 × inner + 0.12 × gate_fraction × e_mean, then × the critical-defect multiplier. `inner` is the weighted sum of the ten non-E tier means (A 0.04, B 0.09, C 0.09, D 0.06, J 0.12, V 0.06, P 0.08, T 0.14, X 0.16, R 0.16 — weights sum to exactly 1.0, and the scorer asserts T+X+R ≥ 0.46 and E ≥ 0.10 so the mechanism axis and the excellence slice cannot be quietly compressed). Each tier mean averages the gamma-transformed scores of that tier's weight-carrying, measured checks: score^gamma_hard for T, X and R, score^gamma_core for the rest (both default 1.0 until calibration, hard-capped at 4.0), and k_P tightens only the TOP rung of each calibration-owned latency ladder. The E (excellence) tier is a separate 0.12 slice multiplied by gate_fraction (the mean unlocked share of 16 named conditions) and e_mean (the mean of the five E-tier checks). The severity model then applies: twelve CRITICAL checks — each one a consequence a correct app provably avoids (crash, data loss, wrong money, dead primary flow) — contribute factor = m + (1−m)·severity with m = 0.6 (calibration-owned), compounding. Severity is measured per consequence class: data-loss fractions cliff to ≤ 0.5 on ANY confirmed silent loss; conservation residuals, lost acknowledged writes and doubled effects cliff to 0; an absent required surface is severity 0 (non-implementation never outscores implemented-with-one-violation). One defect multiplies once, never four times: a critical root that already multiplied suppresses its dependent criticals' multipliers, and rows that are vacuous because a named root already failed price zero through their tier and fire no multiplier of their own. Thresholds load from sb7-thresholds.json; a calibrated file whose sha256 does not match the pin baked into the scorer makes it REFUSE to score, and until the calibration freeze every verdict carries an UNCALIBRATED banner and the version string sb-7.0-rc. The freeze gate runs a monotonicity selftest — synthetic single-defect runs must order wrong-money < data-loss < dead-flow < console-error < minor < cosmetic — and refuses to freeze on any inversion.

Tier weights — the core's composition

A
B
C
D
J
V
P
T
X
R

A Boot & deliverables · 0.04 — the two services boot and bind, the spec-named files exist, the 150 KB asset budget holds, each service owns only its own database

B Wire behaviour · 0.09 — what the API serves on the wire: sync completeness (12,288 rows), money and Berlin-DST bucketing, the append-only event ledger, error envelopes

C Sync discipline · 0.09 — how the vendor is consumed: the 192-page walk, drop resume, Retry-After, the collection-generation rule, webhook and idempotency-key discipline

D Validation & docs · 0.06 — input validation, content types, client timeouts, behavior when the peer service is down, and the DECISIONS.md judgment corners

J Journeys · 0.12 — real user flows in a real browser: first use, the sync click, the maker/checker approval journey, notifications, error and empty states

V Rendered truth · 0.06 — what the page visibly shows: human dates, exponent-correct money, status badges at the frozen hexes, 375 px layout, deliberate styling

P Performance · 0.08 — the spec's budgets, measured: frames under drag, idle flatness (demand rendering), stream apply, API latency under load, sync wall clock

T 3D field · 0.14 — the instanced WebGL field: real context, scene math, GPU pick buffer, camera and coast physics, collision-culled labels, brush, streaming diffs

X Consistency ledger · 0.16 — the live consistency contract: no invented states, per-key version order, monotonic reads, convergence, money conservation, dup/forgery handling

R Resilience · 0.16 — the seeded fault schedule: SIGKILL mid-sync, vendor-down boot, outbox atomicity, partition catch-up, exactly-once effects, workflow durability

E Excellence · 0.12 — the last 12%: drag frames, stream-apply latency, latency under load, optimistic paint, and mastery of the T+X+R mechanisms

The excellence slice — 16 perfection conditions

The last 12% unlocks in proportion to named perfection conditions, then pays out at the excellence tier's own measured mean. Every sb-7 run page shows which conditions that run met.

  • j_first_use == 1.0 — first load renders real data fast, reconciles the on-page total with the fixture, and stays console-clean.
  • j_workflow_journey == 1.0 — the maker/checker approval journey completes end to end through the UI alone.
  • j_error_state == 1.0 — a dead backend produces a visible, actionable error state, never a blank page.
  • j_empty_state == 1.0 — the pre-sync page explains itself per the entrant's own documented D3 decision, never phantom data.
  • console_clean — exactly 0 console errors across the nominal scenarios (load, sync, flow, viz, empty); an unproven scenario fails the condition honestly rather than passing vacuously. The one binary condition.
  • v_responsive_375 ≥ 1.0 — the console works at a phone viewport.
  • v_dates_readable ≥ 1.0 — rendered dates are human-readable, never raw ISO strings.
  • t_scene_binding ≥ 1.0 — the 3D field is bound to the real data, not a decorative canvas.
  • x_conservation_residual ≥ 1.0 — money is conserved: no unexplained minor units created or destroyed.
  • r_no_row_loss ≥ 1.0 — no committed row is missing after any seeded kill.
  • p_drag_frames == 1.0 — the scripted 40-move drag renders at its top frame rung.
  • p_idle_flatness == 1.0 — demand rendering holds: zero draws at rest.
  • p_stream_apply == 1.0 — streamed batches apply within the top budget.
  • p_under_stream == 1.0 — API latency holds its top rung while the SSE stream is live.
  • p_api_latency == 1.0 — the read endpoints hold their top p95 rungs.
  • p_sync_wall == 1.0 — the full 192-page sync completes within the top wall-clock rung.

The critical registry — 12 consequences that multiply the score down

Severity is a property of the consequence, not the tier: each of these contributes factor = 0.6 + 0.4 × measured severity, compounding. Wrong-money and doubled-effect defects are cliffs (severity 0); confirmed silent data loss caps its severity at 0.5.

  • server_runs — crash — the tool does not run
  • sync_completeness — data loss — silently missing payments
  • b_money_rendered — wrong money — wrong exponent/digits or a cross-currency sum
  • b_buckets_dst — wrong money — mis-bucketed days
  • x_conservation_residual — wrong money — unexplained minor units created/destroyed after dupe/loss attribution
  • x_no_lost_write — wrong money — an acknowledged mutation absent from final state
  • r_no_row_loss — data loss — a committed row missing after any seeded kill
  • r_no_dupe_effect — wrong money — a ledger effect applied twice
  • r_cache_truth — data loss — 304-vs-cache mismatch served as fresh
  • r_workflow_durability — data loss — submitted/approved state reverting after SIGKILL
  • j_loads_data — dead primary flow — no data visible
  • j_workflow_journey — dead primary flow — approval cannot complete through the UI

One defect, priced once

One root defect attributes its downstream failures so the report reads as one defect, not many — and, new in sb-7, dedupes the severity multiplier too. Three roots are registered: server_runs (a dead server explains 14 dependents), sync_completeness (a lossy sync explains 23 dependents across B, C, J, T, X and R), and t_vs7dbg_truth (a lying instrumentation layer explains 12 T-tier dependents). Attribution never changes a dependent's score, but a CRITICAL root that already multiplied suppresses its dependent criticals' multipliers — one defect multiplies once, never ×0.6⁴ — and checks that are vacuous because a named root already failed (nothing to lose because nothing loaded) attribute to that root and fire no multiplier of their own.

Diagnostic rows

Two checks are weight-zero diagnostics: j_loads_data and j_console_clean. They are computed and printed on every report but excluded from the J-tier mean because their measurements are absorbed by the compound j_first_use (gate × min of first-data ladder, total reconciliation, console cleanliness). j_loads_data stays in the CRITICAL registry — multiplier-only — because no data visible is a dead primary flow regardless of how the tier means slice it.

Probe failures never blame the app

Harness-attributable absence is never the app's fault: a probe error, a missing vendor-mock surface, or a section lost to a probe timeout marks the row PROBE UNAVAILABLE — excluded from its tier mean, listed loudly in the verdict, never converted into an app zero, and disqualifying for the excellent flag. App-attributable absence is the opposite: a REQUIRED app surface that is missing (no canvas, no drafts panel, no vs7dbg) scores 0 with root attribution, and the severity input for a critical riding it is 0 — an invariant that cannot be evidenced is unproven, and non-implementation must never outscore implemented-with-one-violation.

Worked example

A worked example: inner 0.9614, gate fraction 0.904, e_mean 0.70 composes to 0.88 × 0.9614 + 0.12 × 0.904 × 0.70 = 0.9220. The run's approval journey completed only 5 of 7 parts (j_workflow_journey 0.714, a dead-primary-flow critical), so the critical factor is 0.6 + 0.4 × 0.714 = 0.886, and the published score is 0.9220 × 0.886 = 0.8166 — the top sb-7 baseline, and exactly this arithmetic is printed on its run page.

Fairness: Scoring is serial (one tree, one vendor mock, one pair of app processes per invocation), hermetic (every run deletes leftover graded databases before booting, so the first sync is really a first sync), and served at the vendor port the entrant's own spec advertised. The fixture is seeded per run — collection values, the DST window and the fault-schedule positions all change while the structure and boundaries stay frozen — so memorized constants fail where built-against-structure code passes.

§12 · sb-7 · Every check

Every sb-7 check, piece by piece

All 91 checks the sb-7 scorer runs, grouped by tier. For each: what it measures, the instrument that measures it, what a 1.0 requires, and exactly what lowers it. This documentation is generated from the scorer's source and verified against it — when a run page shows a check at 0.4, the rung that produced 0.4 is written here.

ABoot & deliverablesweight 0.04 · 6 checks

CheckWhat it measures · howEarns 1.0Loses points
a_package_layoutThe submitted tree contains every deliverable the spec names by path — app/__main__.py, DECISIONS.md, and the four web files (index.html, styles.css, app.js, viz.js) — plus a bootable module for each of the two services.Filesystem stat on the 6 named paths, plus a bootable-module test for app.ledgerd and app.notifierd (either app/<svc>.py or app/<svc>/__main__.py counts).All 6 named files exist at their exact paths and both service modules are present — 8/8.Linear: score = present/8, so each missing file or service module costs 0.125; a file at the wrong path or name counts as missing.
server_runsBoth services boot as real processes and report healthy over HTTP within the spec's 10-second budget — the tool runs at all.The harness spawns the services, polls GET / on ledgerd and /health on notifierd, and records ledgerd's time-to-healthy, whether the port was ever bound, and whether the process survived 5 s.GET / answers 200, notifierd reported health at boot, and ledgerd was healthy within 10 seconds.0.75 when both are healthy but ledgerd took longer than 10 s; 0.6 when ledgerd serves but notifierd never reported health (half the tool runs); 0.4 when the port answered but GET / was non-200; 0.15 when the process survives 5 s without binding; 0.0 on crash at boot (the boot error is quoted). CRITICAL: below 0.4 the severity input is 0 and the whole score is multiplied down.
a_combined_entrypointThe documented single-command form — python -m app — really boots BOTH services, not just the two individual service commands the harness normally uses.A separate combined-entrypoint smoke boot: the harness runs python -m app with the full flag set and checks that it boots, that ledgerd reports healthy, and that notifierd reports healthy.All three thirds true: the combined process boots, ledgerd is healthy, notifierd is healthy.Each failing third costs 1/3. PROBE UNAVAILABLE (excluded from the tier mean) when the harness smoke phase never ran.
serves_pageThe backend itself serves the frontend: GET / returns the page and styles.css, app.js, viz.js come back with correct content types, as a real browser would need.HTTP GETs against the running ledgerd: / must return 200 containing markup, then each of the 3 assets passes when it is non-empty AND its live Content-Type contains 'css' or 'javascript' respectively.GET / is 200 with markup and all 3 assets are served non-empty with correct content-type families.0.7 with 2/3 assets correct; 0.4 with 0–1/3 (page still 200 with markup); 0.2 when web/index.html exists on disk and a server answered but / was not a 200 with markup; 0.0 when no server answered at all.
a_asset_budgetThe whole frontend fits the 150 KB source budget and ships zero external code — the budget exists so a vendored 3D library cannot, and the page must work fully offline.stat().st_size sum over the four web files on disk, plus a regex scan of the served page and assets for src=/href= references to http(s) URLs, dynamic import("http…") and fetch("http…") calls.Combined size of the web files present is at most 150 KB (153,600 bytes) AND zero external references — the score is min(size leg, offline leg).Size leg: 0.3 if over budget but within 120% (≤180 KB), 0.0 beyond. Offline leg: a single external reference scores that leg 0, and min() makes it the whole score. 0.0 if no web files exist at all.
a_db_ownershipEach service owns exactly its own SQLite file under --db-dir — ledgerd's ledger.db and notifierd's notifier.db — the one-file-per-service contract.Filesystem inspection of the harness's --db-dir after the run: ledger.db and notifier.db must exist; any other *.db file is counted as an extra.Both ledger.db and notifier.db exist and no extra .db files appear.Each missing database costs 0.5; any extra .db files multiply the result by 0.8. PROBE UNAVAILABLE if the harness never recorded the db-dir.

BWire behaviourweight 0.09 · 11 checks

CheckWhat it measures · howEarns 1.0Loses points
sync_completenessAfter the self-driven full sync the app's local store holds every one of the vendor's 12,288 fixture payments — an incomplete sync silently loses money rows.The best-evidenced full count among GET /api/payments total, the final spot-read total, and GET /api/summary count, capped at 12,288 so run-created payments can never mask missing base rows.An evidenced count of all 12,288 fixture payments.Linear: min(count/12288, 1.0) — 11,000 rows scores ~0.90. Missing fixtures make the row PROBE UNAVAILABLE. CRITICAL (data loss): any shortfall cliffs the severity input to at most 0.5, multiplying the whole score down, and attributes over twenty dependent checks to this one root.
b_total_fieldGET /api/payments reports the correct collection total — a wrong total breaks every caller's paging math.The returned total integer is compared against the fixture count plus the creates committed by probe time (the vendor mid-walk create and approved drafts).total equals the expected at-load collection size exactly.0.75 when total is 12,288 or expected+1 (off by an in-flight create only); 0.5 when within ±4 of expected; 0.0 otherwise. When the sync never produced at least half the collection the row scores 0 as vacuous, attributed to sync_completeness.
b_row_shapeA payment row on the wire carries exactly the 10 documented keys (id, amount_minor, currency, created_at, settled_at, status, version, note, counterparty_name, country) — the vendor's nested counterparty object flattened into the last two.The key set of the first row of GET /api/payments data[] is compared with the frozen 10-key vocabulary.All 10 documented keys present and no undocumented extras.Score is the fraction of the 10 documented keys present; any stray key beyond the documented set subtracts a flat 0.1. An empty data[] scores 0 (attributed to sync_completeness when the sync itself was dead).
b_chronological_orderThe default payments page is ordered by created_at as a parsed instant, not as a string — mixed UTC offsets make string ordering wrong.Every created_at on the default GET /api/payments page is parsed as RFC3339 to an epoch instant and adjacent row pairs are compared numerically.Every adjacent pair is non-decreasing by instant.Score is the fraction of ordered adjacent pairs; a single unparseable created_at anywhere on the page scores the whole check 0 (timestamps are not RFC3339); fewer than 2 rows scores 0.
b_summary_shapeGET /api/summary carries the documented per-currency money blocks: by_currency and the reversals block, both sorted ascending by currency code — the summary is the money surface.Key-and-order inspection of the summary JSON: by_currency rows must carry currency/count/total_minor, be sorted ascending; the reversals block must be present as a list, its rows carry the same three keys, sorted ascending.All five parts: by_currency keys, by_currency sorted, reversals block present, reversals keys, reversals sorted.Each failing part costs 1/5. No summary served at all scores 0 (attributed to sync_completeness when the sync was dead).
b_money_renderedRendered money is arithmetically correct per currency — right digits and right decimal exponent, with the zero-decimal JPY and three-decimal KWD traps weighted separately — and no cross-currency sum exists anywhere in the summary.The load probe harvests rendered amount-cell texts; each cell's detected currency is paired against the fixture's first-page (50-row) amount_minor values — digit-for-digit equality with the minor amount and decimal places equal to the currency exponent. Separately, every top-level integer in /api/summary is scanned for a value equal to the actual sum of the per-currency totals (a real money sum, never a key name).Score = 0.6 × fraction of cells exponent-correct + 0.4 × trap score; full marks need every graded cell correct AND both traps clean (all JPY cells zero-decimal, all KWD cells three-decimal, both currencies rendered), with no cross-currency sum anywhere.Each wrong cell lowers the 0.6 fraction; each trap currency absent or with any wrong cell zeroes its half of the 0.4. Any field carrying the cross-currency sum zeroes the ENTIRE check — a wrong-money cliff, and this check is CRITICAL so it also multiplies the whole score down. No amount cells rendered scores 0; probe error or missing fixture ground truth is PROBE UNAVAILABLE; a dead sync makes the row vacuous, attributed to sync_completeness.
b_buckets_dstGET /api/buckets — the 3D field's aggregate — buckets payments into Europe/Berlin calendar days per status, correct across the seeded DST transition (the data-correctness trap).Every (day, status) → count cell is compared against the fixture's expected buckets; the cell-set size against the expected shape (96 days × 4 statuses = 384 cells at load); the declared timezone against "Europe/Berlin". The DST window — cells within ±2 days of the seeded transition day — is measured separately as the severity leg.All cells exact, cell count matching the expected shape, and timezone == "Europe/Berlin" (weighted 0.7 exact + 0.2 shape + 0.1 timezone).The exact-cell fraction scales the 0.7 slice (an off-by-one on the DST days costs its cells); a wrong cell count halves the shape slice to 0.1; a missing or different timezone loses 0.1; no cells at all scores 0. CRITICAL: the multiplier reads the DST-window cells specifically. Missing fixtures are PROBE UNAVAILABLE; a dead sync makes the row vacuous.
b_viz_recordsGET /api/viz/records — the 3D field's sanctioned full fetch — serves the whole collection columnar with the server-computed Berlin day, so the frontend never recomputes days in UTC.Column inspection of the response: the 7 documented arrays (id, amount_minor, currency, status, created_at, day, version) present as lists; all lengths equal to count; order (created_at instant ASC, id ASC) checked over the first 600 records; each sampled day compared against the fixture's Berlin calendar day for that id.All 7 columns, equal lengths matching count, perfect order, and every sampled day matching the fixture's Berlin day (0.3 columns + 0.2 equal-length + 0.2 order + 0.3 Berlin-day).Each component scales by its measured fraction: missing columns, mismatched lengths, out-of-order pairs and UTC-computed days each drain their slice. Absent or empty endpoint scores 0 (vacuous when the sync was dead); missing fixtures are PROBE UNAVAILABLE.
b_events_logThe append-only event ledger is contiguous from seq 1 with the frozen type and source vocabularies — a seq gap is evidence of a lost write and is graded as one.GET /api/events (bearer-authenticated) is read in full; seqs are compared against range(1, n+1), each event's type against the 8 documented types, source against the 4 documented sources, and required keys (seq, type, at) checked per event.Contiguous seqs from 1 (0.4) plus perfect type vocabulary (0.2), source vocabulary (0.2) and key presence (0.2).A non-contiguous log loses the whole 0.4; vocabulary and key fractions scale their slices. No events with a 404 on the endpoint counts as a required app surface absent (0.0, attributed); any other empty response scores 0.
b_error_envelopeError responses carry the documented structured envelope — error.code and error.message plus field_errors[] with dot-and-[index] paths where validation detail is expected.An HTTP error matrix against the running app; each observed error response is graded for envelope shape (0.6 weight) and, on the cases expecting field errors, for well-formed field paths (0.4 weight).Every observed error response has a well-formed envelope and every field-error-expecting case returns entries with correct string paths and codes.Each half scales by its fraction of passing cases; an envelope without field_errors on validation cases caps at 0.6; no error responses observed at all scores 0.
b_json_shapesEvery sampled API response — success and error paths alike — is parseable JSON; an HTML error page mid-API breaks every client that trusted the contract.An HTTP sample across the API's endpoints; each captured raw body is graded for JSON parseability.All sampled responses parse as JSON.Score is the fraction of responses that parse; no responses sampled scores 0. (Content types are graded separately in d_content_types.)

CSync disciplineweight 0.09 · 7 checks

CheckWhat it measures · howEarns 1.0Loses points
c_paged_walkThe first sync walks the vendor's server-fixed 64-per-page protocol to completion — all 192 pages, documented parameters only, no page fetched twice.The vendor mock's JSONL request trace, sliced to the sync-#1 window: pages served with 200, requests carrying undocumented paging parameters, and duplicate served pages are counted. Compound gate × min over three components.The walk gate holds (at least one page served) and min(pages served/192, 1 − undocumented-param fraction, duplicate-page component) = 1.0 — a complete, clean, single-visit walk.min() rules: an incomplete walk scores its served fraction, each undocumented-param request subtracts 1/list-requests from its component, and each duplicate page costs 10/192 of its component. No list requests at all scores 0 (vacuous, attributed to sync_completeness, when the sync was dead); a missing vendor trace is PROBE UNAVAILABLE.
c_b1_drop_resumeA connection dropped mid-page-stream during the first walk costs a documented resume — not a full restart of committed work, and not a hole in the collection.The vendor mock arms one seeded mid-page connection drop; its ledger records whether the client resumed per the docs, whether it avoided an unconditional full restart, and whether the walk still completed.All three thirds: resumed, no full restart, walk completed.Each failing third costs 1/3. The drop never firing (schedule unreached or vendor surface missing) is PROBE UNAVAILABLE — unless the sync itself was dead, which scores 0 attributed to sync_completeness. No vendor trace is PROBE UNAVAILABLE.
c_b2_retry_afterA seeded 500 with Retry-After during the walk costs exactly one documented retry after the advertised wait — never a fresh unconditional restart of committed work.The vendor mock arms one 500 carrying Retry-After on a walk page; its ledger records whether the client retried exactly once, waited at least the advertised time, and continued the walk from where it was.All three thirds: single retry, waited the advertised Retry-After, continued without restarting committed work.Each failing third costs 1/3. The 500 never firing is PROBE UNAVAILABLE (or a vacuous 0 attributed to sync_completeness when the sync was dead); no vendor trace is PROBE UNAVAILABLE.
c_b5_generation_304The collection-generation rule: a 304 whose X-Collection-Generation disagrees with the stored generation is a cache miss — drop the validator and refetch unconditionally exactly once, without looping and without serving stale data as fresh.The vendor mock arms one lying 304 on a later conditional sync; its ledger counts unconditional refetches, identical conditional repeats, and whether the masked mutation actually propagated to the app's data.All three thirds: exactly one unconditional refetch, at most 3 identical conditional requests (no infinite loop), and the mutation visible locally afterwards.Each failing third costs 1/3 — more than 3 identical conditional requests is the documented infinite-loop bug, and stale-served-as-fresh loses the propagation third (its data-loss half is graded harder in r_cache_truth). Never armed/fired is PROBE UNAVAILABLE; dead sync scores 0 vacuous.
c_conditional_resyncLater syncs are cheap: every re-sync request carries a validator (If-None-Match) except the documented unconditional cases — an unconditional full re-walk on every sync is the expensive-client defect.The vendor trace's post-mutation re-sync windows: requests carrying validators are counted against all re-sync requests, crediting the documented allowed unconditionals (the B5 refetch and first visits to pages that did not exist at load). 304 counts are reported as evidence but not scored — the 304 fraction is vendor-controlled once real mutations move the collection generation.Every re-sync request is conditional (or a documented unconditional): (conditional + allowed)/requests = 1.0.Score is the conditional fraction — a client that re-walks unconditionally scores near 0. No re-sync observed at all scores 0 (vacuous when the sync was dead); no vendor trace is PROBE UNAVAILABLE.
c_webhook_disciplineThe app accepts the vendor's signed push traffic — every delivery acknowledged 2xx within budget — and bounces the one forged signature with 401, state untouched.The vendor mock's delivery ledger: deliveries acknowledged with 2xx are counted against all deliveries, and the forged delivery's recorded HTTP status is read directly.Every delivery 2xx-acked (0.6 weight) and the forged delivery answered exactly 401 (0.4 weight).The acked fraction scales the 0.6; a forged delivery not answered 401 loses the whole 0.4. The vendor delivering no webhooks the app accepted scores 0 (vacuous when the sync was dead); a trace without delivery records, or missing fixtures, is PROBE UNAVAILABLE.
c_send_idempotencyApproved drafts become real vendor payments through POST /v3/payments with a stored Idempotency-Key, and the kill-interrupted send's retry reuses that key — a fresh key per retry is the seeded duplicate-payment bug.The vendor trace's create records: sends carrying an Idempotency-Key header are counted against all sends, and the interrupted send's retry is checked for reusing the same key.Both halves: every send carried a key, and the retry reused the stored key (when no retry was ever needed, the all-keyed evidence stands in).Each failing half costs 0.5. The app never sending an approved draft at all scores 0 — the approve step's SEND half is missing. A trace with no POST /v3/payments records is PROBE UNAVAILABLE.

DValidation & docsweight 0.06 · 5 checks

CheckWhat it measures · howEarns 1.0Loses points
d_content_typesAPI responses declare a JSON content type and the SSE stream declares text/event-stream — a wrong content type breaks strict clients and kills EventSource.The Content-Type headers captured across the API sample, plus the recorded content type of GET /api/stream's response head.Every sampled API response carries a json content type AND /api/stream serves text/event-stream (mean of the two cells).The API cell scales by its fraction of json-typed responses; a wrong or missing SSE content type zeroes that cell; no responses sampled scores 0.
d_validationInvalid input and wrong-role requests are rejected with the documented status codes — bad limit/offset/sort/status 400, unknown paths 404, missing tokens 401, wrong roles 403, self-approval 403 approval_forbidden.An HTTP matrix of invalid requests plus an auth matrix of token/role cases against the running app; each cell compares the observed status code against the documented one.Every cell in both matrices returns exactly the documented status code.Score is the fraction of correct cells (the wrong ones are named in the detail); a matrix that was never exercised scores 0.
d_client_timeoutsThe app is provably resilient to an unresponsive vendor: it boots, binds and serves local data while the vendor refuses connections — one hung vendor call must never hang the tool.Behaviour first: the graded run boots the app inside the seeded vendor-down window (3–8 s) and records whether ledgerd served local state during it; if not proven, a regex grep for timeout= across the backend source (ledgerd.py, meridian.py, vendor.py, sync.py) is the residual.Ledgerd bound and served local state while the vendor refused connections.0.4 residual when the vendor-down boot was not proven but timeout= appears in source; 0.0 with neither — the grep is deliberately demoted to a residual, behaviour outranks it.
d_peer_absenceNeither service crashes on the other's absence: with notifierd down the proxy answers 502 with the documented envelope code and ledgerd keeps running; with ledgerd killed notifierd keeps running.During the seeded partition, GET /api/notifications is read (status + envelope body); liveness of ledgerd through the partition and of notifierd through ledgerd's kill are recorded by the harness.All four quarters: proxy answered 502, envelope code exactly "notifier_unreachable", ledgerd alive through the partition, notifierd alive through the ledger kill.Each failing quarter costs 0.25. The partition window never being orchestrated is PROBE UNAVAILABLE.
d_decisions_docThe three deliberately unstated corners — D1 brush survival on streamed mutation, D2 rejected-draft terminality, D3 pre-first-sync table state — are decided, documented under the frozen headings in DECISIONS.md, and consistent with observed behavior.DECISIONS.md is parsed for ## D1/D2/D3 sections; each documented stance is inferred from its wording and compared against the run's observed behavior (the viz probe's brushed-record mutation for D1, the resubmit attempt's HTTP status for D2, the empty probe's pre-sync state for D3).All three corners documented (at least 15 characters each) with a stance that matches what the run observed.Per corner: 0.0 undocumented, 0.5 documented but consistency not machine-inferable this run, 0.0 documented but contradicted by observed behavior; the score is the mean of the three. Either answer passes — only absence or contradiction fails (§5.7).

JJourneysweight 0.12 · 9 checks

CheckWhat it measures · howEarns 1.0Loses points
j_loads_datadiagWhether the Meridian Payments Console renders any payment rows at all in a real browser, and whether the total it claims on the page matches the collection truth at probe time.The headless-Chromium load probe counts visibly rendered table rows and extracts the DOM-claimed record total; the scorer compares the claimed total against the expected collection size at probe time (the 12,288 fixture payments plus every payment committed by then).At least one visibly rendered data row (0.5) plus a DOM-claimed total exactly equal to the expected total (0.5).Each half is independent: no visible rows loses 0.5; an absent or wrong claimed total loses the other 0.5. A failed load probe is PROBE UNAVAILABLE. Weight-zero diagnostic — absorbed by j_first_use, so the score never enters the Tier J mean — but the check is also CRITICAL ('dead primary flow — no data visible'): a 0 here feeds the multiplier at severity 0 (factor 0.6), while any partial score feeds severity 1.
j_console_cleandiagWhether normal use of the console — loading it and running a sync — produces JavaScript console errors or uncaught page errors.Playwright console/pageerror listeners collect error events across the load and sync scenarios in headless Chromium; the scorer counts the union and quotes the first error text in the detail.Zero console errors and zero uncaught exceptions across both browser sessions.Exactly one error scores 0.5; two or more score 0.0. Weight-zero diagnostic — the measurement is absorbed as j_first_use's console component and as the excellence gate's console_clean condition, so this row does not enter the Tier J mean.
j_first_useThe first-visit experience as one property: real data appears quickly, the on-page total agrees with the collection truth, and the console stays clean.The headless-Chromium load probe stamps time-to-first-data, harvests the DOM-claimed total and console errors; the scorer composes gate × min over three components — a first-data ladder, total reconciliation against the expected count at probe time, and console cleanliness.Gate: at least one visibly rendered data row. Then min() of three components must be 1.0: first data within 2000 ms, the DOM-claimed total exactly equal to the expected total, and zero console errors during the load.Compound gate × min: zero rendered rows scores 0 outright; otherwise the weakest component bounds the whole. The first-data ladder steps 1.0/0.75/0.5/0.25 at 2000/3000/4500/8000 ms (0 beyond); a wrong or missing claimed total sets reconciliation to 0; any console error caps the console component at 0.6 (noise bounds the leg, never zeroes it). A failed load probe is PROBE UNAVAILABLE. Also an excellence-gate condition: only a full 1.0 unlocks its share of the 0.12 slice.
j_sync_journeyThe headline interactive flow: a user clicks Sync now and the app visibly starts, indicates progress, finishes, and refreshes the view.The sync-scenario probe finds and clicks the #sync-now control in headless Chromium and emits causal evidence: button found, an in-flight disabled state, completion, and a changed view (refresh or table-content hash).All four quarter-weighted parts true: button found; an in-flight state (disabled during sync); completed; and the view visibly refreshed (viewRefreshed or a changed table hash).Each missing part costs 0.25: no findable sync button scores 0 ('sync button never found'); a missing in-flight state, an unfinished sync, and an unchanged view each cost their quarter. A failed sync probe is PROBE UNAVAILABLE.
j_workflow_journeyThe maker/checker approval workflow driven end to end through the UI alone: token in, draft created, submitted, approved, listed, notified, and the vendor-created payment landing in the payments table.The flow-scenario probe drives the drafts panel in headless Chromium — enters a role token, fills the draft form, clicks submit and approve — and emits seven causal steps: roleTokenAccepted, draftCreated (a real draft id), submitCausal reached, approveCausal reachedApproved, draftListStates found, notificationSeen, paymentInTable. The scorer composes gate × (steps completed / 7).Both gates hold (the token was accepted and a draft was actually created) and all 7 steps completed through the UI.Gate × fraction: a rejected token or no created draft scores 0 outright; otherwise each missing step costs 1/7. A page with no workflow UI at all (no role token, draft form, or approve affordance) scores 0 as an absent required surface with severity input 0; flow sections lost to the probe's hard cap are PROBE UNAVAILABLE instead. CRITICAL ('dead primary flow — approval cannot complete through the UI'): any shortfall feeds the ×(0.6 + 0.4·severity) multiplier on the whole score. Also an excellence-gate condition.
j_workflow_rejectThe checker's other half of the workflow: rejecting a submitted draft completes through the UI, the rejected state shows in the draft list, and the rejection notification appears in the feed.The same flow-scenario probe clicks #reject-btn on a submitted draft and emits rejectCausal (clicked, reachedRejected, notificationSeen) plus the draft-list rows with their data-state values.All three equal parts: the reject click reached the rejected state, a draft-list row lists state 'rejected', and the reject notification was seen in the feed.Each failed part costs 1/3 ('rejection never completed' when none pass). A failed flow probe is PROBE UNAVAILABLE.
j_notifications_feedThe notifications feed visibly degrades while notifierd is down and heals itself — without a reload — once it returns.The harness SIGKILLs notifierd, lets ledgerd commit further events, then probes the page during the partition and again after the heal, reading #notifications' data-state from each emit and timing how long the feed took to return to 'live'.All four quarter-weighted parts: data-state="degraded" during the partition, data-state="live" after the heal, recovery within 5 seconds of the heal, and no page reload involved.Each missing part costs 0.25 — a feed that never shows degraded, never returns to live, takes longer than 5 s, or only recovers via reload each burn their quarter. Both windows' probes failing is PROBE UNAVAILABLE.
j_error_stateWhat a user sees when the backend is unreachable: a visible, actionable error state instead of a blank or silently broken page.The error-scenario probe loads the page in headless Chromium with the API blocked at the browser network layer and scans for a visible error element whose text matches actionable phrasing (try again / retry / check / running / refresh / vendor / offline).A visible error element with actionable phrasing; when the probe also exercised retry (emitted retryRecovered), the retry must actually recover.Ladder: 0.6 for visible + actionable where an exercised retry did not recover; 0.3 for a visible error indication without actionable text; 0.0 for no error state at all. The retry rung binds only when the probe emitted it. A failed error probe is PROBE UNAVAILABLE. Also an excellence-gate condition.
j_empty_stateWhat a fresh install looks like before the first sync completes: an honest empty-or-progress state rather than phantom data or a blank page.The scorer boots a second instance against a brand-new empty database and probes it in headless Chromium, reading rendered rows, the pre-sync state the page declares (the DECISIONS.md D3 corner — empty-with-progress vs blocked), and any visible empty-state text.Zero rendered data rows plus either a rendered pre-sync state consistent with the app's documented D3 decision or visible empty-state text.Any rendered row on the empty database scores 0 (phantom data). A page that renders content but has no empty state scores 0.3; a blank page scores 0.0. A failed empty probe is PROBE UNAVAILABLE. Also an excellence-gate condition.

VRendered truthweight 0.06 · 5 checks

CheckWhat it measures · howEarns 1.0Loses points
v_dates_readableWhether the Date column shows humans a readable date instead of a raw machine timestamp.The load probe captures the rendered text of date cells from headless-Chromium computed rendering; the scorer regex-classifies the strings.At least one date cell rendered, no cell containing raw ISO-8601 (YYYY-MM-DDTHH:MM), and at least one recognizably human text (month letters, or a d/m or d.m numeric pattern).No date cells rendered scores 0; any raw ISO-8601 timestamp shown to users scores 0 (the spec forbids machine timestamps in the rendered page); formatted-but-not-recognizably-locale-readable text scores 0.5. A failed load probe is PROBE UNAVAILABLE. Also an excellence-gate condition.
v_money_presentationWhether rendered amounts identify their currency — the presentation half of money; the exponent-and-digits truth is graded separately by the critical b_money_rendered.The load probe harvests the rendered amount-cell texts; the scorer counts the fraction carrying a recognizable currency token (€/EUR, $/USD, ¥/JPY, KWD/د.ك).Every rendered amount cell carries a recognizable currency token.Score is the tagged fraction of rendered amount cells — an amount with no currency is ambiguous money. No amount cells rendered scores 0. A failed load probe is PROBE UNAVAILABLE.
v_status_badgesWhether the four payment statuses are visually distinguishable at a glance and painted in the spec's frozen palette — the same four hexes the 3D field uses.The probe reads computed color and background of each status badge on page one; the scorer compares against the frozen hexes — settled #059669, pending #D97706, refunded #7C3AED, failed #B91C1C — with a ±8-per-channel tolerance on either the background or the text color, and checks every (color, background) pair is unique.Score = 0.6 × (statuses at their frozen hex / max(3, statuses rendered)) + 0.4 × distinctness; full marks need at least 3 rendered statuses, every one at its frozen hex, and all style pairs unique.No status cells rendered scores 0. Each status off its frozen hex lowers the 0.6 component; fewer than 3 rendered statuses or any two statuses sharing an identical style pair zeroes the 0.4 distinctness component. A failed load probe is PROBE UNAVAILABLE.
v_responsive_375Whether the page survives a phone-width viewport: no horizontal scrolling and real content still rendered at 375 px.The probe resizes the headless-Chromium viewport to 375 px, reloads, and compares document scroll width against the viewport while re-counting visibly rendered rows.No horizontal scroll and at least one rendered row at 375 px; when the probe measures tap targets (emits tapTargetsOk) they must also be adequately sized — otherwise full credit stands on no-scroll plus rows, with the detail noting tap targets were not measured.Any horizontal scroll at 375 px scores 0.0 (the page breaks on a phone). No scroll but zero rendered rows also scores 0.0 — empty pages never scroll, so the pass would be vacuous. Small tap targets, when measured, cap the score at 0.6. A failed load probe is PROBE UNAVAILABLE. Also an excellence-gate condition.
v_stylingWhether the page is deliberately styled rather than browser-default: a real stylesheet, layered surface colors, a chosen font, and a branded header.The probe inspects the rendered page for a stylesheet, counts distinct computed non-transparent background colors, reads the body's computed font-family, and checks for a visible #app-header (falling back to a source grep of the shipped HTML when the probe did not emit the field).All four weighted parts: stylesheet present (0.3), at least 3 distinct background colors (0.25), a body font that is set and not Times (0.25), and a branded #app-header (0.2).Each absent part loses its weight independently: no stylesheet −0.3, fewer than 3 distinct backgrounds −0.25, empty or Times-default font −0.25, no visible app header −0.2. A failed load probe is PROBE UNAVAILABLE.

PPerformanceweight 0.08 · 6 checks

CheckWhat it measures · howEarns 1.0Loses points
p_drag_framesThe 3D field stays interactive under input: real frames rendered during the scripted 40-move budget drag at the full 12,288-instance count.The viz probe's GL wrapper counts frames over the scripted 40-move drag (first move → pointerup + one rAF); the scorer grades the frame delta against calibration-owned rungs.At least 32 frames over the drag (rungs are calibration-owned; pre-freeze defaults shown).≥24 frames earns 0.75, ≥16 earns 0.5, ≥8 earns 0.25, fewer 0. Unmeasurable frames score 0. A viz probe error or the drag section lost to the probe's hard cap is PROBE UNAVAILABLE. Below 1.0 also locks this check's share of the excellence gate.
p_idle_flatnessDemand rendering, the frozen spec rule: at rest — no input, no coast, no pending stream batch — the scene draws nothing; a continuous rAF render loop fails by design.The viz probe samples 500 ms rest windows and counts default-framebuffer draw calls in each through the GL wrapper, excluding windows where a stream batch landed; the scorer takes the worst window. This rule is frozen in the spec (0 draws per 500 ms at rest), not calibration-owned.Zero default-framebuffer draws in the worst sampled 500 ms rest window.Binary: any draw call in any clean rest window scores 0. No idle windows sampled scores 0 ('demand rendering unproven'). A viz probe error or a section lost to the hard cap is PROBE UNAVAILABLE. Below 1.0 also locks this check's share of the excellence gate.
p_stream_applyA live SSE batch becomes visible fast: the median time from receipt to applied — store, digest and pixels — against the spec's 250 ms budget.The viz probe times every graded stream batch's apply window; the scorer takes the median applyMs and grades it on calibration-owned rungs (top rung tightened by k_P once calibrated).Median apply within 250 ms (250/k_P when calibrated).≤500 ms earns 0.75, ≤1000 ms 0.5, ≤2000 ms 0.25, slower 0. No stream batch ever applied scores 0 ('the live stream never landed'). A viz probe error or the stream section lost to the hard cap is PROBE UNAVAILABLE. Below 1.0 also locks this check's share of the excellence gate.
p_under_streamThe API stays fast AND correct while the SSE stream burst is landing — read p95 under proven concurrent load.Harness-side reader threads issue GET /api/payments requests while the vendor fires a stream burst; every sample is timestamped so the overlap fraction is a measured fact, and every response must be a correct, well-formed page.Overlap fraction ≥ 0.5, every response correct, and p95 ≤ 150 ms (calibration-owned rungs, top tightened by k_P).p95 ≤300 ms earns 0.75, ≤600 ms 0.5, ≤1200 ms 0.25. ANY wrong or empty response under load scores 0 — fast wrong answers are not performance. If fewer than half the reads provably overlapped the burst, or the measurement never ran, the check REFUSES (PROBE UNAVAILABLE) rather than crediting or zeroing on unproven load. Below 1.0 also locks this check's share of the excellence gate.
p_api_latencyThe read endpoints answer within their spec budgets when idle: the worst p95 across the API latency battery.A harness-side battery of repeated GETs against the read endpoints in the idle phase; the scorer takes the WORST endpoint's p95 and grades it on calibration-owned rungs (spec budgets: /api/payments and /api/summary 150 ms, /api/buckets 200 ms, /api/viz/records 400 ms).Worst idle p95 ≤ 150 ms (150/k_P when calibrated).≤300 ms earns 0.75, ≤600 ms 0.5, ≤1500 ms 0.25, slower 0. The battery never running is PROBE UNAVAILABLE. Below 1.0 also locks this check's share of the excellence gate.
p_sync_wallWall-clock time for the self-driven first sync — the full 192-page walk, seeded faults and documented waits included — against the spec's 120 s budget.The harness wall-times sync #1 from the app's own unprompted boot-time walk, faults (dropped connection, 500 + Retry-After) included.Sync #1 completes within 120 s (120000/k_P ms when calibrated).≤240 s earns 0.75, ≤420 s 0.5, ≤600 s 0.25, slower 0. A sync #1 that never completed inside the harness budget scores 0. Below 1.0 also locks this check's share of the excellence gate.

T3D fieldweight 0.14 · 15 checks

CheckWhat it measures · howEarns 1.0Loses points
t_context_realThe #viz3d panel is a real, drawing WebGL surface with the pinned context attributes and a correctly sized backing store — not a styled div, an image, or an unused canvas.Headless-Chromium viz probe with a wrapped GL layer: it verifies a webgl/webgl2 context on the canvas with id viz3d, reads the context-creation attributes the app asked for, compares the backing store to clientWidth × devicePixelRatio, counts wrapped default-framebuffer draw calls, and samples a blind pixel grid for non-background coverage.All five weighted parts: a WebGL context on #viz3d (0.25), creation attributes exactly {antialias: false, alpha: false} (0.15), backing store matching rect × DPR (0.20), at least 1 real draw call (0.15), and grid coverage with ≥ 3 non-background samples in ≥ 2 distinct colors (0.25).Weighted sum — each failed part drops its weight: a context on the wrong canvas loses 0.25, unpinned attributes 0.15, a mis-sized backing store 0.20, zero draws 0.15, a canvas painted in one flat color fails the 0.25 coverage part. A viz probe error, or the contextReal section lost to the probe's hard timeout, is PROBE UNAVAILABLE — excluded from the tier mean.
t_layout_basisThe locked layout basis vs7dbg.layout() reports — d0 (first day), D0 = 96 (the span), R0 (max in-day count at load) — matches the fixture truth and never moves when a streamed create arrives.The probe calls vs7dbg.layout() and the scorer compares each of d0/D0/R0 against the fixture pack's independently computed basis; the stream scenario re-reads layout() after a streamed create and checks it is unchanged.All three basis cells exact, and the basis unchanged after the streamed create.Score is the fraction of the three cells that match (each worth 1/3); a basis that MOVED after a streamed create caps the whole check at 0.25 — the spec pins the basis at load precisely because the vendor value-dates creates in-span. Missing fixtures, a probe error, or the layout section lost to the probe timeout are PROBE UNAVAILABLE.
t_scene_bindingThe rendered scene actually encodes the payment data: vs7dbg.sceneDigest()'s seven statistical moments over all 12,288 instanced columns match an independent recomputation from the fixture.The probe reads sceneDigest() — {count, Sh, Sh2, Sx, Sz, Sxh, Szh} in float64 — and the scorer compares each moment against its own recomputation of the frozen §3.1 transform (Δ = 1.2 pitch, log10 height law through the currency exponent), tolerance |Δ| ≤ max(0.5, 1e-4·|expected|).All 7 digest moments within tolerance.Anything short of 7/7 pays only 0.5 × the matching fraction — a single wrong cell in 12,288 exceeds the tolerance, so partial credit is deliberately halved. If the sync never completed the row scores 0 as vacuous (attributed to sync_completeness, no multiplier of its own). Missing fixtures, probe error or a timed-out digest section are PROBE UNAVAILABLE.
t_height_pixelsColumn heights are true in rendered pixels — including the JPY (exponent 0) and KWD (exponent 3) instances whose heights expose a forgotten currency exponent.At a close-up pose the probe measures rendered column tops in device pixels for a set of sampled instances and checks each against the analytic height h = clamp(0.9 + 0.55·log10(a_major), 0.2, 4.2) within ±3 px, plus the top-face status color.Every measured case within ±3 px with the correct top color, AND both JPY and KWD instances among the measured currencies.Score is the fraction of cases passing both the ±3 px and color tests; if JPY and KWD are not both present among the measured cases the whole score is multiplied by 0.7 — the exponent trap must actually be exercised. No measured cases scores 0. A dead sync makes the row vacuous (0, attributed to sync_completeness); probe error or a lost section is PROBE UNAVAILABLE.
t_draw_budgetThe field renders 12,288 instances inside the draw budget — at most 8 default-framebuffer draw calls per rendered frame — forcing instanced draws or a merged buffer instead of per-column draws.The GL wrapper counts every default-framebuffer draw call over the scripted 40-move budget drag (window: first move → pointerup + one rAF) and checks the delta against BOTH limits: ≤ 8 × max(frames drawn, 1) and ≤ 8 × (40 + 8) = 384.At least one draw observed and the drag's draw-call delta within both limits, i.e. never more than 8 default-FBO draws per rendered frame.Zero draws over the window scores 0; drawing but exceeding either limit scores 0.25 — over-budget draws mean per-instance rendering, the exact technique the budget forbids. Probe error or the dragBudget section lost to the timeout is PROBE UNAVAILABLE.
t_pick_bufferPicking is GPU truth: the offscreen pick buffer answers occlusion exactly as the depth buffer says, on click points constructed to kill CPU raycasts and last-drawn-wins shortcuts.The probe grades pick points where vs7dbg.pick(), the raw vs7dbg.pickPixel() bytes, and the scorer's analytically computed front instance must all three agree — including four adversarial constructions: a bar occluded by a lower-index bar, by a higher-index bar, a partial occlusion, and a background point inside the field's convex hull.Every graded pick agrees three ways (0.6 weight) and all four occlusion constructions have at least one passing case (0.4 weight).Score = 0.6 × agreeing-pick fraction + 0.4 × (passing constructions / 4). Last-drawn-wins fails one of the two index-order constructions by design; a CPU raycast that ignores the depth buffer fails the partial-occlusion case. No graded pick points scores 0; a dead sync is vacuous (0, attributed to sync_completeness); probe error or a lost picks section is PROBE UNAVAILABLE.
t_pick_real_passThe pick buffer is real GPU work with the documented cost profile: a fresh offscreen pass after each scene invalidation, bounded at 4 offscreen draws, and never flashing ID colors on the visible canvas.The GL wrapper's counters around the probe's pick calls: since the last scene invalidation the first pick must be preceded by ≥ 1 offscreen draw AND ≥ 1 offscreen readPixels; the refresh must cost ≤ 4 offscreen draws; and pick calls must cause 0 default-framebuffer draws.All four counter conditions hold (the score is the MIN of the four binary parts).min() of binaries — any one failure scores 0: no offscreen draw or no readback since invalidation (a CPU raycast with fabricated pickPixel bytes), more than 4 offscreen draws per refresh, or any visible-canvas draw during a pick. No pick-refresh counter windows observed scores 0. Probe error or a timed-out section is PROBE UNAVAILABLE.
t_click_semanticsCanvas clicks follow the frozen §3.3 semantics: click an instance to toggle it into the brush, click it again to toggle it out, click background to clear the brush.Real synthetic pointer input in headless Chromium during the brush exercise: the probe clicks an instance and reads vs7dbg.brush() (toggle in), clicks it again (toggle out), then clicks a background pixel and checks the set emptied.All three: instance click toggles in, the same click toggles out, background click clears.Each failing part costs 1/3. Click semantics never exercised at all scores 0. Probe error or the brush section lost to the timeout is PROBE UNAVAILABLE.
t_camera_mathThe orbit camera implements the documented math exactly: defaults yaw 30 / pitch 40 / distance 260, the 0.30 deg-per-px drag law, the exponential wheel law without page scroll, both clamps, double-click reset, and a projection that matches the printed formula.Scripted input verified against recomputation: default pose read within 0.5; drag yaw error ≤ 1.0°; two wheel steps each matching distance · exp(0.0012·deltaY) within max(0.5, 1%) with no page scroll; pitch driven to its 85° clamp (±0.5); a distance-clamp hit; double-click restoring all three defaults (±0.5) with angular velocity zeroed; and vs7dbg.project() spot-checked against the probe's own projection with max error ≤ 3 px.All seven parts: defaults, drag law, wheel law, pitch clamp, distance clamp, double-click reset, projection.Score is the passing fraction of the 7 equal parts — a sign-flipped drag, a wheel that scrolls the page, a reset that keeps coast velocity, or projection error over 3 px each cost 1/7. Camera math never exercised scores 0; probe error or a timed-out section is PROBE UNAVAILABLE.
t_coast_identityPost-release inertia obeys the closed-form τ = 0.4 s decay law: at any coasting instant the remaining travel equals v(t)·τ, a slow release starts no coast, and the coast settles inside the printed budget.After a scripted flick the probe samples vs7dbg.camera() mid-coast and the scorer checks the remaining-coast identity yaw_rest − yaw(t) = v(t)·τ per sample within tolerance max(1.0°, 0.15·|v·τ|); a separate drag ending below 6 px/s must drift ≤ 0.5°; settle time is compared to τ·ln(max(v0,2)/2) + 0.7 s capped at 2.5 s.Every mid-coast residual within tolerance (0.5 weight), the slow release coast-free (0.25), and the settle budget met (0.25).Score = 0.5 × identity fraction + 0.25 × slow-release + 0.25 × settle. A per-frame constant decay tuned for 60 Hz drifts measurably and fails the identity samples. No coast evidence at all scores 0; probe error or a lost coast section is PROBE UNAVAILABLE.
t_coast_realityThe coast is real motion on the canvas, not a camera() narrative: after a fast flick the scene provably keeps moving past release.After a ≥ 600 px/s flick the probe requires yaw to travel ≥ 3° past release in the drag direction, and corroborates with pixel spot-checks: mid-coast framebuffer samples must actually change, and the rest pose must land where the projection says.Coasted at least 3° (0.4), in the drag direction (0.2), with every pixel spot-check confirming movement (0.4).Weighted sum: no 3° travel loses 0.4, wrong direction 0.2, and the pixel fraction scales the remaining 0.4 — camera() alone can lie, so unmoved pixels earn nothing. No flick-coast evidence scores 0; probe error or a lost section is PROBE UNAVAILABLE.
t_labels_cullingThe 12 highest-amount records get screen-space labels that are collision-culled deterministically THROUGH the app's own pick buffer — exact set, exact geometry, zero overlap, never floating over an occluded instance.At a decisive pose the scorer recomputes the expected label set (priority a_major DESC, id ASC; eligible iff the anchor projects on-canvas AND pick(anchor) returns that instance; greedy rect-intersection culling) and compares the DOM's #viz-labels children: the shown set, each 110 × 18 px border-box (±1 px), the (+10, −9) anchor offset (±2.5 px), data-id attributes, pairwise overlap, and that no shown label sits on a pick-ineligible instance.Exact shown-set match (0.35), all rects at 110 × 18 (0.15), all offsets within ±2.5 px (0.1), data-id on every shown label (0.1), zero overlap violations (0.15), and no label on an occlusion-culled instance (0.15).Weighted sum — a wrong set loses the 0.35 outright, geometry/offset/data-id scale by their per-label fractions, any overlapping pair zeroes the 0.15 overlap part, any label over a pick-hidden instance zeroes the 0.15 occlusion part. No decisive pose graded scores 0; a dead sync is vacuous (0, attributed to sync_completeness); probe error or a lost labels section is PROBE UNAVAILABLE.
t_brush_linkOne brush set links the table and the 3D field in both directions: row clicks and instance clicks toggle the same set, non-members dim to the exact 0.30 pixel rule, the count readout tracks, and background click restores full color.The probe drives both doors with real clicks and reads nine facts: table-row click toggles membership; non-member pixels at round(0.30·c) and member pixels at full status hex (framebuffer readback, ±8/channel); #brush-count present and rendering text; instance click toggles; the table navigated to the clicked record's page with the row in the viewport; the row carrying data-brushed="true"; background click emptying the set; and pixels restored to full hex.All nine parts true.Score is the passing fraction of the 9 equal parts; neither door working at all scores 0. A dead sync is vacuous (0, attributed to sync_completeness); probe error or a lost brush section is PROBE UNAVAILABLE.
t_stream_diffSSE batches apply as true diffs: only the changed instances upload, no buffer realloc, the digest moves by exactly the change, and the changed instance's pixels show it.The GL wrapper accounts every uploaded buffer byte during each batch-apply window against the budget |S| × 64-byte stride + 4096 and counts reallocs (bufferData > 4096 bytes); the scorer also checks the post-batch sceneDigest() delta against recomputation and a framebuffer probe of the changed instance.Every batch within its byte budget (0.35), zero reallocs (0.15), every graded digest delta correct (0.3), and the changed-instance pixel check passing (0.2).Weighted sum: the byte and digest parts scale by their per-batch fractions; any realloc in any window zeroes the 0.15; a failed pixel check zeroes the 0.2. A full-array re-upload per message is exactly what the byte budget catches. No SSE batch observed applying scores 0; probe error or a lost stream section is PROBE UNAVAILABLE.
t_vs7dbg_truthThe mandated window.vs7dbg instrumentation tells the truth: camera() agrees with the pixels, sceneDigest() with the recomputed data, frames() with the wrapper's counted draws, and pick() with pickPixel() with the analytic answer.In-page evaluation cross-checks four legs: camera defaults within 0.5° yaw error with a confirming rest-pose pixel; the digest against the scorer's recomputation; frames() against wrapper-observed drawing; and the pick/pickPixel/analytic triplet agreement.All four legs: camera-vs-pixels (0.3), digest-vs-recompute (0.3), frames-vs-wrapper (0.2), pick-triplet (0.2).Weighted sum — each lying leg drops its weight. window.vs7dbg absent entirely is a required app surface missing: scores 0 with app attribution (never PROBE UNAVAILABLE), and as a ROOT_BLOCKS root it attributes the shortfalls of 12 dependent T-tier checks to this one defect. Probe error or a timed-out section is PROBE UNAVAILABLE.

XConsistency ledgerweight 0.16 · 12 checks

CheckWhat it measures · howEarns 1.0Loses points
x_l1_no_invented_statesEvery (payment, version) the app ever applied or served exists in the vendor's committed history — an invented state means the app fabricated data.The scorer replays the app's event log and its sampled read stream against the committed version set derived from the fixture pack plus the scheduled delivery script, cross-checked against the vendor surface.Zero invented (payment, version) pairs across all observations.Binary cliff: a single invented state scores 0 — there are no partial rungs on fabrication. No observations at all scores 0; a dead sync is vacuous (0, attributed to sync_completeness); missing fixtures are PROBE UNAVAILABLE.
x_l2_per_key_orderApplied versions per payment strictly increase in event-log order — duplicate and stale webhook outcomes belong in the counters, never as events.The scorer walks the app's ledger events (payment.created / payment.updated), tracking the last version per payment id and counting same-key pairs where the version failed to increase.Zero order violations over all same-key event pairs.With violations the score is 1 − violations/pairs, so each out-of-order or re-applied event costs its share. No events scores 0; a dead sync is vacuous (0, attributed to sync_completeness).
x_l3_monotonic_readsThe version served for a payment never decreases from one read to the next — a sync page landing after a webhook applied v+1 must not regress the row.Throughout the live run the scorer samples the app's own API into a read stream and compares consecutive observed versions per payment id.Zero regressions over all sampled read pairs.Binary cliff: any regression scores 0 — this is the buffered-blind-upsert failure mode and there is no partial credit. An empty read stream scores 0; the scorer's read stream never running is PROBE UNAVAILABLE; a dead sync is vacuous (0, attributed to sync_completeness).
x_l4_convergenceAt quiescence every payment's version and status equals the vendor's final committed state, and the row count equals the vendor's — the mid-walk create present exactly once.Post-everything spot reads of the touched payments are compared per-id against the fixture-derived final state, and the app's final total against N = 12,288 plus every payment the run itself created.Every spot-read row at the vendor-final version AND status (0.7) and the final total exact (0.3).Score = 0.7 × converged-row fraction + 0.3 × (total exact ? 1 : 0). No final spot reads scores 0; a dead sync is vacuous (0, attributed to sync_completeness); missing fixtures are PROBE UNAVAILABLE.
x_l5_group_atomicityNo read ever observes half a transaction group: the refunded payment and its reversal become visible together or not at all.The scorer's read stream deliberately samples summaries through the refund window; each snapshot is checked for a confirmed half-applied state (refunded row visible without its reversal total, or the reverse).Zero confirmed half-applied observations across all samples.Binary cliff: one confirmed half state scores 0 — atomicity has no partial rungs. The read stream never sampling the refund window is PROBE UNAVAILABLE; a dead sync is vacuous (0, attributed to sync_completeness).
x_m1_amount_immutabilityNo served row ever shows an amount_minor different from the vendor-committed amount — v3 never mutates amounts, only status/note/version.Every amount observed in the read stream and the final spot reads is compared against the fixture's committed amount for that payment id.Zero mutated amounts across all observations.Binary cliff: a single mutated amount scores 0. No amounts observed scores 0; a dead sync is vacuous (0, attributed to sync_completeness); missing fixtures are PROBE UNAVAILABLE.
x_m2_pair_conservationAt every observed instant, per currency, the summary's reversal totals equal the refunded rows' amounts — both halves of each refund visible, or neither.Each read-stream summary snapshot through the refund window is checked for pair conservation; the same sampling that grades L5 supplies the evidence.Every snapshot pair-conserved with zero confirmed half states.Any confirmed half-applied state scores 0 outright; otherwise the score is the conserved fraction of snapshots. No snapshots scores 0; the refund window never sampled is PROBE UNAVAILABLE; a dead sync is vacuous (0, attributed to sync_completeness).
x_m3_terminal_conservationTerminal per-currency counts and totals — reversals included — equal vendor ground truth: fixture plus scripted mutations plus every payment the app created.The final /api/summary is compared bucket-by-bucket against the fixture's expected by_currency map (count and total_minor exact per currency) and against the expected reversals map including the scripted refund's reversal.Every currency bucket exact (0.7) and every reversal bucket exact (0.3).Score = 0.7 × exact-currency fraction + 0.3 × exact-reversal fraction — each wrong bucket costs its share. A dead sync is vacuous (0, attributed to sync_completeness); missing fixtures are PROBE UNAVAILABLE.
x_m4_no_cross_currencyNo field anywhere in the summary carries a cross-currency money sum — minor units are not a common denomination and summing them is wrong money.The scorer computes the actual cross-currency sum of the by_currency totals and scans every top-level integer field of both graded summary snapshots for a value equal to it (a legitimate single-currency total never matches).No field equal to the cross-currency sum in any scanned summary.Binary cliff: any offending field scores 0. No summary to scan scores 0.
x_conservation_residualCRITICAL — after every duplicate and loss is attributed, no minor units remain created or destroyed: the money conservation residual is zero in every currency.Per currency: residual = served final total − expected total + attributed losses − attributed duplicates, using the run's own dupe/loss attribution so a loss plus an equal-value duplicate can no longer net to zero.Residual exactly 0 in every currency after attribution.Binary cliff: any unexplained residual (or an unreadable total) scores 0, and as a critical its severity input is 0 — the score multiplier takes the full 0.6-floor hit. No terminal summary scores 0. A dead sync makes the row vacuous: 0, attributed to sync_completeness, firing no multiplier of its own. Missing fixtures are PROBE UNAVAILABLE.
x_no_lost_writeCRITICAL — every mutation the app acknowledged with a 2xx is present in the final state; an acked-then-vanished write is silent data loss.The vendor trace's acknowledged webhook deliveries are checked against the app's final rows: each acked (payment, version) must be reflected by a final version at least that high.Zero lost acknowledged mutations.Binary cliff: one lost write scores 0, with critical severity input 0 (full multiplier hit). No checkable acked deliveries against final rows scores 0; a trace without delivery-ack records is PROBE UNAVAILABLE; a dead sync is vacuous (0, attributed to sync_completeness, no own multiplier).
x_ooo_dup_forgedThe three webhook trust traps land correctly: the out-of-order pair keeps v+2 (the late v+1 never overwrites), the forged-signature event changes nothing, and no duplicate delivery applies twice.The scorer reads the schedule's seeded targets — the out-of-order pair's payment, the forged event's payment and note — from the app's final rows against the fixture-final truth, and scans the event log for any (payment, version) applied twice.All observable parts true: out-of-order target at the final version with no stale note, forged target untouched (right version, forged note absent), and zero duplicate event applications.Score is the passing fraction of the observable parts (up to 3, equal weight). None of the targets observable scores 0; a dead sync is vacuous (0, attributed to sync_completeness); missing fixtures are PROBE UNAVAILABLE.

RResilienceweight 0.16 · 10 checks

CheckWhat it measures · howEarns 1.0Loses points
r_b3_sigkill_resyncledgerd SIGKILLed mid-sync restarts, converges, and duplicates nothing — a kill mid-walk costs a clean cursor restart, never dupes or holes.The harness kills ledgerd after the n-th list response of the graded walk, restarts it with the same flags, and grades three facts from the vendor trace and the app's post-restart state: it restarted, it converged to the committed collection, and no row was duplicated.All three: restarted, converged, no duplicates.Each failing part costs 1/3. The B3 kill window never orchestrated is PROBE UNAVAILABLE; the kill unreached because sync #2 never produced its n-th list response is vacuous when the sync is dead (0, attributed to sync_completeness), otherwise PROBE UNAVAILABLE with the reason.
r_b4_vendor_down_bootThe app boots with the vendor down: binds within 10 seconds anyway, serves local data, does not crash, and completes the first sync unprompted once the vendor returns.The harness holds the vendor down for the seeded 3–8 s boot window and grades four facts: bound within 10 s, served requests while the vendor was down, no crash, and recovered without any operator action.All four parts true.Each failing part costs 1/4 — a boot that waits for the vendor loses bound_in_10s, a crash on vendor absence loses no_crash, a sync that needs a manual nudge loses recovered_unattended. The window never armed is PROBE UNAVAILABLE.
r_b6_outbox_atomicA SIGKILL between an outbox commit and its delivery loses nothing and doubles nothing — the exact window where commit-then-POST and POST-then-commit both fail.The harness kills ledgerd after an outbox row commits but before the relay delivers it, restarts, and grades: the row was pending before the kill, the relay resumed after restart, delivery was exactly-once, and no event was lost.All four parts: pending before kill, resumed, delivered exactly once, none lost.Each failing part costs 1/4 — a dual-write app typically loses the event (none_lost) or delivers it twice (exactly_once). The window never arranged is PROBE UNAVAILABLE.
r_b7_partitionA notifier partition degrades visibly and heals in order: writes never block, /api/outbox/status reports down with growing pending, the feed shows degraded, and catch-up is in seq order with the UI live again within 5 seconds of heal.The harness SIGKILLs notifierd, commits 8 further ledgerd events during the outage, restarts it, and grades five facts from the API, the relay's delivery order, and the load probe's feed states.All five: writes never blocked, status reported down+pending, UI showed degraded, catch-up in seq order, feed live within 5 s of heal without a reload.Each failing part costs 1/5 — a user write blocking on the dead notifier, out-of-order catch-up, or a feed that needs a reload each burn their fifth. The partition never orchestrated is PROBE UNAVAILABLE.
r_notifier_exactly_onceThe notifier's durable processed set proves exactly-once: every outbox-crossing event processed, no seq twice, surviving kills.The scorer reads /notify/processed after the run and grades: the seq set is duplicate-free, it covers every expected outbox-crossing seq, and the notifier's health counters expose a duplicate count.Unique seqs (0.5) with full crossing coverage (0.4) and the duplicate counter present (0.1).A duplicated seq zeroes the 0.5; coverage scales the 0.4 by the fraction of expected crossing seqs present; a missing duplicate counter drops 0.1. An empty durable set after a run that exercised the partition scores 0; the set never read at all is PROBE UNAVAILABLE.
r_notification_multisetSelective materialization is exact: draft.submitted, draft.approved, draft.rejected and reversal.created each produce exactly one notification row; payment.sent produces none.The scorer compares the notifier's notification rows, counted by kind, against the expected multiset computed from the app's own committed ledger events (fixture fallback when the ledger is empty), and separately counts leaked payment.sent rows.Every kind's count exact and zero payment.sent notifications.Score is the exact-kind fraction over the union of expected and observed kinds; any leaked payment.sent row halves the whole score. Notify-everything and notify-nothing both fail. The notifications endpoint never read, or no committed events to derive the expectation from, is PROBE UNAVAILABLE.
r_no_row_lossCRITICAL — no committed row goes missing after any seeded kill: the row count after every SIGKILL+restart is at least the count before it.The kill ledger records the app-reported payments count immediately before and after every seeded kill (B3, B6, the workflow kills); the scorer compares each pair.Every kill survived with after ≥ before.Binary cliff: any kill losing rows scores 0, with critical severity input capped at 0.5 by the data-loss transform — the multiplier hit is unavoidable. Vacuous-pass gate: an app that never had rows earns nothing for not losing them — the row scores 0 attributed to sync_completeness with no own multiplier. The kill matrix never running is PROBE UNAVAILABLE.
r_no_dupe_effectCRITICAL — no ledger effect applies twice: one vendor payment per approved draft, no duplicate notification rows, no (type, payment, version) event effect repeated.Three legs of evidence: the vendor's count of payments created for the graded approval (must be ≤ 1 — the idempotency-key replay test), duplicate event_seq values among notification rows, and repeated payment.updated/payment.sent event keys in the app's ledger.Every observable leg clean.Binary cliff: any duplicated effect scores 0, with critical severity input 0 (full multiplier hit) — a doubled vendor payment is wrong money. No dupe-effect evidence gathered is PROBE UNAVAILABLE; a dead sync is vacuous (0, attributed to sync_completeness, no own multiplier).
r_cache_truthCRITICAL — the lying-304 trap: a 304 whose X-Collection-Generation disagrees with the stored generation is a cache miss, and serving the stale cache as fresh is graded data loss.The harness arms exactly one 304 with a mismatched generation header during a later conditional sync (B5), commits mutations behind it, and checks whether the app's post-B5 state reflects them or serves the stale rows as fresh.The post-304 state reflects the committed mutations — the app dropped its validator and refetched unconditionally, exactly once.Binary cliff: stale served as fresh scores 0, with the data-loss severity transform capping the critical input at ≤ 0.5. B5 never armed, or the post-B5 snapshot missing, is PROBE UNAVAILABLE; a dead sync is vacuous (0, attributed to sync_completeness, no own multiplier).
r_workflow_durabilityCRITICAL — submitted and approved are durable the moment their 200 is written: a SIGKILL immediately after either, including mid-send, must find the state intact after restart.The harness SIGKILLs ledgerd immediately after a draft submit's 200 (A1) and again immediately after an approve's 200 with the vendor send still in flight (A2), restarts, and reads each draft's state: A1 must still be submitted (or legitimately later: approved/sent), A2 must be approved or sent.Both kill placements find the acknowledged state intact.Binary cliff: any reverted state scores 0, with the data-loss severity transform capping the critical input at ≤ 0.5. Drafts endpoints absent entirely is a required app surface missing — scores 0 with app attribution and severity input 0. The workflow exercise never running, or neither kill placement firing, is PROBE UNAVAILABLE.

EExcellenceweight 0.12 · 5 checks

CheckWhat it measures · howEarns 1.0Loses points
e_frames_under_dragExcellence-grade fluidity: frames rendered during the scripted 40-move drag at the full 12,288-instance count, with proof the frames actually drew.The same viz drag instrument as p_drag_frames, but graded on tighter excellence rungs and requiring a positive default-framebuffer draw-call delta — a frame counter that ticks with zero draws (an empty rAF loop) earns nothing.At least 40 frames over the drag with a positive draw-call delta (calibration-owned rungs).≥32 frames earns 0.75, ≥24 earns 0.5, ≥12 earns 0.25, fewer 0. Unmeasurable frames or zero draw calls score 0. Probe error or a lost drag section is PROBE UNAVAILABLE. Like every E row, the payout is further scaled by the proportional excellence gate (fraction of the named perfection conditions met) before the 0.12 slice pays.
e_stream_apply_latencyExcellence-grade streaming: the median SSE batch apply time against rungs 2.5× tighter than the P-tier budget.The same median applyMs instrument as p_stream_apply, graded on the excellence rungs (top tightened by k_P once calibrated).Median apply within 100 ms.≤200 ms earns 0.75, ≤400 ms 0.5, ≤800 ms 0.25, slower 0. No batch applied scores 0. Probe error or a lost stream section is PROBE UNAVAILABLE. Scaled by the proportional excellence gate before the 0.12 slice pays.
e_under_load_latencyThe read API's p95 under the proven stream-burst load, graded again in the excellence slice.The same under-stream measurement as p_under_stream: reader threads racing the SSE burst, overlap measured per sample, correctness required on every response.Overlap ≥ 0.5, every response correct, p95 ≤ 150 ms (calibration-owned rungs).p95 ≤300 ms earns 0.75, ≤600 ms 0.5, ≤1200 ms 0.25. Any wrong/empty response under load scores 0. An overlap below the 0.5 floor — or a measurement that never ran — REFUSES as PROBE UNAVAILABLE (unproven load licenses nothing). Scaled by the proportional excellence gate before the 0.12 slice pays.
e_optimistic_paintThe workflow UI paints optimistically: the submitted state appears while the write is provably still on the wire, then really saves.Causal proof in the flow probe: the write's network response is held, the probe confirms the UI painted during the hold (paintedWhileHeld), stamps the paint latency page-side, and checks the state actually saved after the hold releases.Painted-while-held (the causal gate, 0.5 base) + paint stamp ≤ 100 ms (full 0.25 latency term; calibration-owned rungs) + the state confirmed saved after release (0.25).No optimistic exercise in the flow emit, or a UI that waits for the network, scores 0. Paint at 100–250 ms earns 0.6 of the latency term, 250–800 ms 0.3, slower 0; a failed save drops its 0.25. Flow evidence lost to the probe's hard cap is PROBE UNAVAILABLE. Scaled by the proportional excellence gate before the 0.12 slice pays.
e_masteryExcellence includes the mechanisms, not only the surface: mastery of the entire 3D contract (T), consistency-and-money invariants (X) and resilience (R) tiers together.Computed from the scorer's own already-graded T, X and R rows in the same run — the mean over every row that was measurable and not vacuous (rows attributed to an already-failed root are excluded).T+X+R mean ≥ 0.90.Below the 0.90 cliff the score is (mean/0.90) × 0.5 — near-mastery pays at most half. No measurable T/X/R rows at all is PROBE UNAVAILABLE. Scaled by the proportional excellence gate before the 0.12 slice pays.