Claude Sonnet 5 — 0.9692 on Gauntlet 5.2

Checks that the five files the spec names by path exist: meridian.py, store.py, api.py, __main__.py, web/index.html. Scored as the fraction present.

5/5 named files

Parses meridian.py and store.py and checks the eight named methods are declared on MeridianClient (fetch_all_payments, total_count, create_payment) and Store (upsert_many, all_payments, count, last_sync, set_last_sync). Scored as the fraction present.

8/8 methods

Starts python -m vendorsync on a free port and polls GET /api/health for up to 25 s; 1.0 when the server binds and answers 200.

healthy

GET / must return 200 with HTML served by the backend; 1.0 when it does.

GET / -> 200

GET /api/health must carry the three documented keys status, payments, last_sync. Scored as the fraction present.

3/3 documented keys

POST /api/sync must return the three documented keys fetched, inserted, total. Scored as the fraction present.

3/3 documented keys

After one sync, the reported total is compared with the fixture's 247 payments; scored as the fraction retrieved.

247/247 payments after one sync

A second POST /api/sync must report inserted=0 with the total unchanged at 247; 1.0 when idempotent, 0.5 when the total is right but rows were re-inserted.

second sync inserted=0 total=247

Three requests against /api/payments: default limit must return 25 rows, limit=500 must cap at 100, limit=5&offset=5 must return 5 rows. Scored one third each.

default/cap/offset: yes/yes/yes

The first payment row must carry exactly the documented keys id, amount_minor, currency, created_at, status; undocumented extra keys deduct 0.2.

['amount_minor', 'created_at', 'currency', 'id', 'status']

GET /api/payments must report total equal to the fixture's 247 payments; 1.0 when exact.

total=247

Parses created_at across the first page and counts adjacent pairs in non-decreasing instant order; scored as the ordered fraction.

24/24 adjacent pairs ordered

GET /api/summary total_minor is compared with the fixture sum; 1.0 when exact, decreasing linearly with relative error.

total_minor=4409197 (want 4409197)

Summary oldest/newest must be present (0.4), carry a UTC designator — Z or +00:00 (0.3), and be in order (0.3).

oldest=2026-03-01T12:00:00+00:00 newest=2026-03-07T08:36:00+00:00

A matrix of invalid requests (negative or non-numeric limit/offset, unknown paths) must return the documented status and error body; each cell scores half for the status, half for the body.

6.0/6 cells clean

Greps the served page source for loading, empty and error state markup; one third per state present.

states present: ['loading', 'empty', 'error']

Greps the page source for currency formatting: Intl.NumberFormat or toLocaleString scores 1.0; manual /100 or toFixed(2) scores 0.5.

Intl formatting

The served page must reference no external http(s) assets; any CDN src or href scores 0 (the spec requires offline operation).

0 external asset(s)

Reads all rows as raw wire bytes across three pages: documented keys (0.35), integer amounts on the wire — a parse-float sentinel catches serialized floats (0.25), parseable timestamps (0.15), id coverage of the 247-payment collection (0.25).

247 rows: keys 247/247, int-amounts 247/247, times 247/247, ids 247/247

Orders every fetched row against the vendor's true instant order and scores the fraction of adjacent pairs ordered correctly — 246 pairs of resolution.

246/246 adjacent pairs in true instant order

Samples six success and error paths; each response scores half for parsing as JSON and half for declaring an application/json content type.

6.0/6 responses fully JSON

Graded in quarters: status ok, a fresh boot reporting 0 payments and null last_sync, a post-sync count of 247, and a UTC-designated last_sync.

ok=ok fresh=True post_count=247 last_sync=2026-08-15T10:05:32.036196+00:00

The vendor request trace must contain a fetch of the vendor's /docs endpoint; 1.0 when the documentation was actually read.

3 docs fetches

The trace must show cursor-based paging on /v1/payments — the documented scheme — rather than offset-style parameters.

cursor=True offset-style=False over 25 calls

The client must paginate to exhaustion, reaching the page whose next_cursor is null; stopping early scores 0.

9 pages, offsets [0, 100, 200], reached end: True

The vendor answers 429 with Retry-After in seconds; the trace must show the client waiting at least that long before retrying.

waited 2.01s, Retry-After said 2s

Retry-After also arrives as an RFC 7231 HTTP-date; the client must parse the date form and wait accordingly.

retried +0.01s relative to the stated moment

The vendor expires a cursor with 410; the documented recovery is to restart from the first page, not to treat 410 as the end of data.

restarted from page 0: True; re-sent the dead cursor first: False

MeridianClient.fetch_all_payments(), invoked directly, must return all 247 fixture payments.

returned 247, expected 247

MeridianClient.total_count() must return the collection size (247), not the size of one page.

total_count() returned 247, expected 247

The client's returned list must be in true instant order despite mixed UTC offsets in created_at.

chronological

create_payment() replayed with the same idempotency key receives the vendor's documented 409 replay answer; the client must treat it as success and return the same payment id.

first='pay_new_0000' replay='pay_new_0000'

Every payment-creating POST in the trace must carry an Idempotency-Key header.

2/2 POSTs carried Idempotency-Key

Amounts returned by the client must be integers in minor units; any float scores 0.

0 non-integer amounts of 247

The vendor mutates 25 payment statuses, a third sync runs, and the local store is paged to count how many mutated statuses arrived; scored as the fraction propagated.

25/25 mutated statuses visible locally after sync3

The app is SIGKILLed and restarted on the same --db; the row count, read through the app's own API, must survive. Scored as the surviving fraction.

247/247 rows survived kill+reboot on the same db

The second sync's vendor requests are inspected for conditional refetch: every page answered 304 scores 1.0, all-conditional but refetched 0.7, mechanism present 0.4, absent 0.

9/9 conditional, 9/9 answered 304

Vendor requests issued for the first sync, compared with the measured optimum of 7 (three pages at limit=100 plus the unavoidable trap chain); scored as optimum/actual.

7 vendor requests for the first sync (optimum 7)

The largest limit parameter the client ever requested, divided by the documented maximum of 100.

largest limit requested: 100 (max allowed 100)

Two overlapping POST /api/sync calls run in parallel; the payment count must still be exactly 247 afterwards.

total after two overlapping syncs: 247

/api/payments must declare Content-Type: application/json.

Content-Type: 'application/json'

Greps the page source for five operator affordances: last-sync shown, control disabled while syncing, row count, retry affordance, accessible table headers. One fifth each.

5/5: ['last-sync shown', 'disables while syncing', 'row count', 'retry affordance', 'accessible table']

Reads store.py: the payments write must use ON CONFLICT … DO UPDATE (a true merge). INSERT OR REPLACE/IGNORE scores 0.5, select-then-insert 0.3.

ON CONFLICT DO UPDATE (true merge)

store.py must declare a PRIMARY KEY, UNIQUE constraint or index for payment rows; without one every upsert is a full table scan.

keyed

meridian.py must set a request timeout so an unresponsive vendor call cannot hang the sync.

no request timeout

The page's error text must tell the user what to do (retry, check the connection); the bare word 'error' scores 0.4.

actionable guidance

A headless browser loads the page: half for rendering at least one data row, half for the DOM reconciling with the 247-payment collection.

25 rows rendered, DOM claims 247 of 247

Console errors collected across the load and sync journeys: zero scores 1.0, one 0.5, more 0.

0 console error(s) across load+sync

The browser finds the sync control and clicks it; four equal parts: button found, disabled while syncing, completed, view refreshed.

button_found, disabled_while_syncing, completed

With the API blocked, the page must show a visible error state with actionable text.

error state shown: Could not load payments: Failed to fetch Retry (api blocked, bodyText 152)

A second instance on a fresh database must show a visible empty state and render zero rows.

empty state shown: No payments yet. Click “Sync now” to pull data from Meridian

Date cells read from the rendered page: raw ISO-8601 shown to users scores 0; locale-readable text scores 1.0.

rendered dates like Mar 1, 2026, 02:00 PM

Status cells must render with visually distinct styles per status; a single unstyled status scores 0.

only 'SETTLED' rendered on page 1; badge-styled

Pagination controls present (0.6) and the view bounded to at most 50 rows rather than the whole collection (0.4).

controls=True, 25 rows in one view (spec: never all 247)

A status filter control must be present in the rendered page.

status filter control present

The page is loaded at a 375 px viewport; any horizontal scrolling scores 0.

375px horizontal scroll: False

Rendered styling: a stylesheet applied (0.4), at least three distinct background colors (0.3), a non-default font (0.3).

stylesheet=True, 3 distinct backgrounds, font '-apple-system, "Segoe UI", Roboto, Helve'

GET /api/payments with limit=25, measured over repeated requests; 1.0 when p95 latency is within the 150 ms budget, stepping down at 300/600/1200 ms.

GET /api/payments limit=25 p95 0.6678749341517687 ms (budget 150)

Time from navigation until the first data rows render; 1.0 within the 2000 ms budget, stepping down at 3000/4500/8000 ms.

first data rows rendered at 33 ms (budget 2000)

POST /api/sync wall-clock time; 1.0 within the 60 s budget, stepping down at 90/120/180 s.

POST /api/sync wall 3.4968748223036528 ms (budget 60000)