Cloud baseline
GPT-5.6 Terra — 0.3252 on Gauntlet 6.0
openai.gpt-5.6-terra · scorer sb-6.0
- Overall
- 0.3252
- Excellent
- no
- Benchmark
- Gauntlet 6.0
- Recorded duration
- 6m 30s
Tier breakdown
Scoring detail
How this score was built
Per-check scorer output, as posted by the app. Expand a section to inspect its checks.
The number, exactly
(0.88 × 0.8247 core + 0.12 × 0.54 gate × 0.75 excellence) × 0.4200 critical = 0.3252
Critical defects compound a multiplier on the whole score (pre-severity 0.7743):
The core (88% of the total) is the weighted mean of the nine measured tiers. The last 12% is the excellence slice: it unlocks in proportion to the perfection conditions below (7 of 14 met here), then pays out at the excellence tier's own measured mean. Core 0.7257 + excellence 0.0485.
| Component | Weight | Earned | Points |
|---|---|---|---|
| Core (tiers A–D) | 60% | — | — |
| A · Structure | 25% of core | 1.00 | 0.150 |
| B · Behaviour | 30% of core | 1.00 | 0.180 |
| C · Vendor contract | 25% of core | 1.00 | 0.150 |
| D · Finesse | 20% of core | 1.00 | 0.120 |
| Journey (J) | 15% | 0.388 | 0.058 |
| Visual (V) | 10% | 0.171 | 0.017 |
| Performance (P) | 5% | 0.833 | 0.042 |
| Hard blocks | 10% | — | — |
| Total | 0.1169 | ||
files, modules and interfaces named by the spec exist; the server starts and binds
modules present
1.00Checks that the five files the spec names by path exist: meridian.py, store.py, api.py, __main__.py, web/index.html. Scored as the fraction present.
8/8 named files
interfaces declared
1.00Parses meridian.py and store.py and checks the eight named methods are declared on MeridianClient (fetch_all_payments, total_count, create_payment) and Store (upsert_many, all_payments, count, last_sync, set_last_sync). Scored as the fraction present.
15/15 methods
server runs
1.00Starts python -m vendorsync on a free port and polls GET /api/health for up to 25 s; 1.0 when the server binds and answers 200.
healthy in 0.4s
serves page
1.00GET / must return 200 with HTML served by the backend; 1.0 when it does.
page 200, 3/3 assets served with correct content types
a asset budget
1.0051 KB of 150 KB budget (4/4 files present)
health shape
1.00GET /api/health must carry the three documented keys status, payments, last_sync. Scored as the fraction present.
4/4 top keys, 5/5 webhook counters
sync shape
1.00POST /api/sync must return the three documented keys fetched, inserted, total. Scored as the fraction present.
4/4 documented keys
HTTP requests against the running app; responses compared to expected values
sync completeness
1.00After one sync, the reported total is compared with the fixture's 247 payments; scored as the fraction retrieved.
1553/1553 payments after full sync
total field
1.00GET /api/payments must report total equal to the fixture's 247 payments; 1.0 when exact.
total=1553 (want 1553)
payment row shape
1.00The first payment row must carry exactly the documented keys id, amount_minor, currency, created_at, status; undocumented extra keys deduct 0.2.
10/10 documented keys
chronological order
1.00Parses created_at across the first page and counts adjacent pairs in non-decreasing instant order; scored as the ordered fraction.
49/49 adjacent pairs ordered by instant
summary bounds utc
1.00Summary oldest/newest must be present (0.4), carry a UTC designator — Z or +00:00 (0.3), and be in order (0.3).
oldest=2026-03-22T23:00:00Z newest=2026-04-05T21:56:00Z
b summary currency
1.004/4 currency buckets exact
b buckets dst
1.0056/56 cells exact, shape 56/56, tz=Europe/Berlin
b error envelope
1.00envelope 1.00, field paths 1.00 over 7 cases
row integrity
1.00Reads all rows as raw wire bytes across three pages: documented keys (0.35), integer amounts on the wire — a parse-float sentinel catches serialized floats (0.25), parseable timestamps (0.15), id coverage of the 247-payment collection (0.25).
1553 rows: keys 1553/1553, int-amounts 1553/1553, times 1553/1553, ids 1553/1553
json everywhere
1.00Samples six success and error paths; each response scores half for parsing as JSON and half for declaring an application/json content type.
6.0/6 responses fully JSON
ui offline
1.00The served page must reference no external http(s) assets; any CDN src or href scores 0 (the spec requires offline operation).
0 external asset ref(s)
local pagination
1.00Three requests against /api/payments: default limit must return 25 rows, limit=500 must cap at 100, limit=5&offset=5 must return 5 rows. Scored one third each.
default50/cap200/offset: [True, True, True]
input validation
1.00A matrix of invalid requests (negative or non-numeric limit/offset, unknown paths) must return the documented status and error body; each cell scores half for the status, half for the body.
7.0/7 status codes correct
resync idempotent
1.00A second POST /api/sync must report inserted=0 with the total unchanged at 247; 1.0 when idempotent, 0.5 when the total is right but rows were re-inserted.
second sync inserted=0 total=1553
vendor API usage checked against the vendor documentation: pagination, ordering, idempotent create
vendor traps
1.00{'retry_after': 1.0, 'cursor_expiry': 1.0, 'stall': 1.0}
vendor cursor paging
1.00The trace must show cursor-based paging on /v1/payments — the documented scheme — rather than offset-style parameters.
73/86 cursor requests, 0 undocumented-param requests
c batch partial
1.00{'order_kept': True, 'succeeded_once': True, 'failed_reported': True, 'no_retry': True}
c api depth
1.00schema=1.00 pagination=1.00 validation=1.00
update propagation
1.00The vendor mutates 25 payment statuses, a third sync runs, and the local store is paged to count how many mutated statuses arrived; scored as the fraction propagated.
12/12 mutated statuses visible locally after sync3
restart persistence
1.00The app is SIGKILLed and restarted on the same --db; the row count, read through the app's own API, must survive. Scored as the surviving fraction.
1553/1553 rows survived kill+reboot
output formats, edge cases and error shapes the spec requires
api content type
1.00/api/payments must declare Content-Type: application/json.
6/6 JSON content types
client timeouts
1.00meridian.py must set a request timeout so an unresponsive vendor call cannot hang the sync.
sync survived the vendor stall within bound
uses max limit
1.00The largest limit parameter the client ever requested, divided by the documented maximum of 100.
largest vendor page size requested: 100
store indexed
1.00store.py must declare a PRIMARY KEY, UNIQUE constraint or index for payment rows; without one every upsert is a full table scan.
keyed
store atomic upsert
1.00Reads store.py: the payments write must use ON CONFLICT … DO UPDATE (a true merge). INSERT OR REPLACE/IGNORE scores 0.5, select-then-insert 0.3.
ON CONFLICT DO UPDATE
concurrent sync safe
1.00Two overlapping POST /api/sync calls run in parallel; the payment count must still be exactly 247 afterwards.
total after two overlapping syncs: 1553
ui error actionable
1.00The page's error text must tell the user what to do (retry, check the connection); the bare word 'error' scores 0.4.
actionable text: The activity data failed to load. Check that the server is r
a headless browser loads the page, triggers a sync and reads the rendered table
j loads data
0.00A headless browser loads the page: half for rendering at least one data row, half for the DOM reconciling with the 247-payment collection.
0 rows rendered, DOM claims None
j console clean
0.00Console errors collected across the load and sync journeys: zero scores 1.0, one 0.5, more 0.
3 console error(s) across load+sync: TypeError: Cannot read properties of undefined (reading 'mount')
j first use
0.00ttfd=Nonems reconcile=0.0 consoleErrs=2
j sync journey
0.25The browser finds the sync control and clicks it; four equal parts: button found, disabled while syncing, completed, view refreshed.
button_found
j error state
1.00With the API blocked, the page must show a visible error state with actionable text.
visible + actionable: The activity data failed to load. Check that the server is r
j empty state
0.30A second instance on a fresh database must show a visible empty state and render zero rows.
page not blank but no empty state
rendered-page checks: stylesheet served, data rows present, mobile layout at 375px
v dates readable
0.00Date cells read from the rendered page: raw ISO-8601 shown to users scores 0; locale-readable text scores 1.0.
no date cells rendered
v currency rendered
0.00no amount cells rendered
v status distinct
0.00Status cells must render with visually distinct styles per status; a single unstyled status scores 0.
no status cells rendered
v pagination
0.00Pagination controls present (0.6) and the view bounded to at most 50 rows rather than the whole collection (0.4).
controls=False, 0 rows in one view
v filter
0.20A status filter control must be present in the rendered page.
filter control present; its options were not drivable
v responsive 375
0.00The page is loaded at a 375 px viewport; any horizontal scrolling scores 0.
no h-scroll but zero rendered rows at 375px (vacuous)
v styling
1.00Rendered styling: a stylesheet applied (0.4), at least three distinct background colors (0.3), a non-default font (0.3).
stylesheet=True, 3 backgrounds, font '"Avenir Next", "Segoe UI", sys', header=True
response-time budgets measured against the running app
p list latency
1.00GET /api/payments with limit=25, measured over repeated requests; 1.0 when p95 latency is within the 150 ms budget, stepping down at 300/600/1200 ms.
GET /api/payments limit=50 p95 1.8219160847365856 ms (budget 150)
p buckets latency
1.00GET /api/buckets p95 3.831499954685569 ms (budget 200)
p summary latency
1.00GET /api/summary p95 0.510042067617178 ms (budget 150)
p page interactive
0.00Time from navigation until the first data rows render; 1.0 within the 2000 ms budget, stepping down at 3000/4500/8000 ms.
first data rows rendered at None ms (budget 2000)
p first frame
1.00first non-background 3D frame at 22.6 ms (budget 3000)
p sync wall
1.00POST /api/sync wall-clock time; 1.0 within the 60 s budget, stepping down at 90/120/180 s.
POST /api/sync wall 8.3512500859797 ms (budget 90000, documented waits included)
Run notes and corrections
Cloud baseline — a single goose run session on the frozen VendorSync Pro spec, scored by the sb-6 scorer with the severity model (critical defects — crash, wrong money, data loss, dead primary flow — compound a multiplier on the composed score). Serial, hermetic, advertised vendor port. NOTE: this session was terminated early by a goose engine defect (compaction shipped reasoning content to Bedrock; fixed in 19b4ed6ef) — the score is a floor for this model, not a ceiling.
Screenshots
Recorded views of the built application. Captions identify the available captures; the number of images does not imply a number of repair rounds.
Run details
- Model
- openai.gpt-5.6-terra