Claude Sonnet 5 — 0.4971 on sb-5.3

Checks that the five files the spec names by path exist: meridian.py, store.py, api.py, __main__.py, web/index.html. Scored as the fraction present.

5/5 named files

Parses meridian.py and store.py and checks the eight named methods are declared on MeridianClient (fetch_all_payments, total_count, create_payment) and Store (upsert_many, all_payments, count, last_sync, set_last_sync). Scored as the fraction present.

8/8 methods

Starts python -m vendorsync on a free port and polls GET /api/health for up to 25 s; 1.0 when the server binds and answers 200.

healthy

GET / must return 200 with HTML served by the backend; 1.0 when it does.

GET / -> 200

GET /api/health must carry the three documented keys status, payments, last_sync. Scored as the fraction present.

3/3 documented keys

POST /api/sync must return the three documented keys fetched, inserted, total. Scored as the fraction present.

0/3 documented keys

After one sync, the reported total is compared with the fixture's 247 payments; scored as the fraction retrieved.

0/247 payments after one sync

A second POST /api/sync must report inserted=0 with the total unchanged at 247; 1.0 when idempotent, 0.5 when the total is right but rows were re-inserted.

second sync failed

Three requests against /api/payments: default limit must return 25 rows, limit=500 must cap at 100, limit=5&offset=5 must return 5 rows. Scored one third each.

default/cap/offset: no/yes/no

The first payment row must carry exactly the documented keys id, amount_minor, currency, created_at, status; undocumented extra keys deduct 0.2.

no rows

GET /api/payments must report total equal to the fixture's 247 payments; 1.0 when exact.

total=0

Parses created_at across the first page and counts adjacent pairs in non-decreasing instant order; scored as the ordered fraction.

too few rows

GET /api/summary total_minor is compared with the fixture sum; 1.0 when exact, decreasing linearly with relative error.

total_minor=0 (want 4409197)

Summary oldest/newest must be present (0.4), carry a UTC designator — Z or +00:00 (0.3), and be in order (0.3).

oldest=None newest=None

A matrix of invalid requests (negative or non-numeric limit/offset, unknown paths) must return the documented status and error body; each cell scores half for the status, half for the body.

6.0/6 cells clean

Greps the served page source for loading, empty and error state markup; one third per state present.

states present: ['loading', 'empty', 'error']

Greps the page source for currency formatting: Intl.NumberFormat or toLocaleString scores 1.0; manual /100 or toFixed(2) scores 0.5.

Intl formatting

The served page must reference no external http(s) assets; any CDN src or href scores 0 (the spec requires offline operation).

0 external asset(s)

Reads all rows as raw wire bytes across three pages: documented keys (0.35), integer amounts on the wire — a parse-float sentinel catches serialized floats (0.25), parseable timestamps (0.15), id coverage of the 247-payment collection (0.25).

no rows on the wire

Orders every fetched row against the vendor's true instant order and scores the fraction of adjacent pairs ordered correctly — 246 pairs of resolution.

too few known rows to order

Samples six success and error paths; each response scores half for parsing as JSON and half for declaring an application/json content type.

6.0/6 responses fully JSON

Graded in quarters: status ok, a fresh boot reporting 0 payments and null last_sync, a post-sync count of 247, and a UTC-designated last_sync.

ok=ok fresh=True post_count=0 last_sync=None

The vendor request trace must contain a fetch of the vendor's /docs endpoint; 1.0 when the documentation was actually read.

0 docs fetches

The trace must show cursor-based paging on /v1/payments — the documented scheme — rather than offset-style parameters.

cursor=False offset-style=False over 0 calls

The client must paginate to exhaustion, reaching the page whose next_cursor is null; stopping early scores 0.

0 pages, offsets [], reached end: False

The vendor answers 429 with Retry-After in seconds; the trace must show the client waiting at least that long before retrying.

the API never throttled — client did not reach page 2

Retry-After also arrives as an RFC 7231 HTTP-date; the client must parse the date form and wait accordingly.

never reached the date-form throttle

The vendor expires a cursor with 410; the documented recovery is to restart from the first page, not to treat 410 as the end of data.

never reached the expiring cursor

MeridianClient.fetch_all_payments(), invoked directly, must return all 247 fixture payments.

returned 247, expected 247

MeridianClient.total_count() must return the collection size (247), not the size of one page.

total_count() returned 247, expected 247

The client's returned list must be in true instant order despite mixed UTC offsets in created_at.

chronological

create_payment() replayed with the same idempotency key receives the vendor's documented 409 replay answer; the client must treat it as success and return the same payment id.

first='pay_new_0000' replay='pay_new_0000'

Every payment-creating POST in the trace must carry an Idempotency-Key header.

2/2 POSTs carried Idempotency-Key

Amounts returned by the client must be integers in minor units; any float scores 0.

0 non-integer amounts of 247

The vendor mutates 25 payment statuses, a third sync runs, and the local store is paged to count how many mutated statuses arrived; scored as the fraction propagated.

mutation pass not exercised or sync3 absent

The app is SIGKILLed and restarted on the same --db; the row count, read through the app's own API, must survive. Scored as the surviving fraction.

no rows before the kill — nothing to prove persistence with

The second sync's vendor requests are inspected for conditional refetch: every page answered 304 scores 1.0, all-conditional but refetched 0.7, mechanism present 0.4, absent 0.

second sync absent or reported nothing

Vendor requests issued for the first sync, compared with the measured optimum of 7 (three pages at limit=100 plus the unavoidable trap chain); scored as optimum/actual.

no vendor requests observed

The largest limit parameter the client ever requested, divided by the documented maximum of 100.

largest limit requested: none (max allowed 100)

Two overlapping POST /api/sync calls run in parallel; the payment count must still be exactly 247 afterwards.

total after two overlapping syncs: 0

/api/payments must declare Content-Type: application/json.

Content-Type: 'application/json'

Greps the page source for five operator affordances: last-sync shown, control disabled while syncing, row count, retry affordance, accessible table headers. One fifth each.

4/5: ['last-sync shown', 'disables while syncing', 'row count', 'accessible table']

Reads store.py: the payments write must use ON CONFLICT … DO UPDATE (a true merge). INSERT OR REPLACE/IGNORE scores 0.5, select-then-insert 0.3.

select-then-insert

store.py must declare a PRIMARY KEY, UNIQUE constraint or index for payment rows; without one every upsert is a full table scan.

keyed

meridian.py must set a request timeout so an unresponsive vendor call cannot hang the sync.

no request timeout

The page's error text must tell the user what to do (retry, check the connection); the bare word 'error' scores 0.4.

actionable guidance

A headless browser loads the page: half for rendering at least one data row, half for the DOM reconciling with the 247-payment collection.

0 rows rendered, DOM claims 0 of 247

Console errors collected across the load and sync journeys: zero scores 1.0, one 0.5, more 0.

1 console error(s) across load+sync: Failed to load resource: net::ERR_EMPTY_RESPONSE

The browser finds the sync control and clicks it; four equal parts: button found, disabled while syncing, completed, view refreshed.

button_found, disabled_while_syncing

With the API blocked, the page must show a visible error state with actionable text.

error state shown: Could not reach vendorsync backend: Failed to fetch (api blocked, bodyText 109)

A second instance on a fresh database must show a visible empty state and render zero rows.

empty state shown: No payments yet. Click "Sync now" to pull payments from Meri

Date cells read from the rendered page: raw ISO-8601 shown to users scores 0; locale-readable text scores 1.0.

no date cells rendered

Status cells must render with visually distinct styles per status; a single unstyled status scores 0.

no status cells rendered

Pagination controls present (0.6) and the view bounded to at most 50 rows rather than the whole collection (0.4).

controls=False, 0 rows in one view (spec: never all 247)

A status filter control must be present in the rendered page.

no status filter control

The page is loaded at a 375 px viewport; any horizontal scrolling scores 0.

375px horizontal scroll: False

Rendered styling: a stylesheet applied (0.4), at least three distinct background colors (0.3), a non-default font (0.3).

stylesheet=True, 2 distinct backgrounds, font '-apple-system, "Segoe UI", Helvetica, Ar'

GET /api/payments with limit=25, measured over repeated requests; 1.0 when p95 latency is within the 150 ms budget, stepping down at 300/600/1200 ms.

GET /api/payments limit=25 p95 0.34133298322558403 ms (budget 150)

Time from navigation until the first data rows render; 1.0 within the 2000 ms budget, stepping down at 3000/4500/8000 ms.

first data rows rendered at None ms (budget 2000)

POST /api/sync wall-clock time; 1.0 within the 60 s budget, stepping down at 90/120/180 s.

POST /api/sync wall 1.5323329716920853 ms (budget 60000)