Cloud baseline
Claude Haiku 4.5 — 0.4615 on Gauntlet 5.3
us.anthropic.claude-haiku-4-5-20251001-v1:0 · scorer sb-5.3
- Overall
- 0.4615
- Hard
- 0.17
- Excellent
- no
- Benchmark
- Gauntlet 5.3
Tier breakdown
Scoring detail
How this score was built
Per-check scorer output, as posted by the app. Expand a section to inspect its checks.
| Component | Weight | Earned | Points |
|---|---|---|---|
| Core (tiers A–D) | 60% | 0.468 | 0.281 |
| A · Structure | 25% of core | 0.83 | 0.125 |
| B · Behaviour | 30% of core | 0.36 | 0.066 |
| C · Vendor contract | 25% of core | 0.10 | 0.015 |
| D · Finesse | 20% of core | 0.63 | 0.075 |
| Journey (J) | 15% | 0.650 | 0.098 |
| Visual (V) | 10% | 0.333 | 0.033 |
| Performance (P) | 5% | 0.667 | 0.033 |
| Hard blocks | 10% | 0.167 | 0.017 |
| Total | 0.4615 | ||
files, modules and interfaces named by the spec exist; the server starts and binds
modules present
1.00Checks that the five files the spec names by path exist: meridian.py, store.py, api.py, __main__.py, web/index.html. Scored as the fraction present.
5/5 named files
interfaces declared
1.00Parses meridian.py and store.py and checks the eight named methods are declared on MeridianClient (fetch_all_payments, total_count, create_payment) and Store (upsert_many, all_payments, count, last_sync, set_last_sync). Scored as the fraction present.
8/8 methods
server runs
1.00Starts python -m vendorsync on a free port and polls GET /api/health for up to 25 s; 1.0 when the server binds and answers 200.
healthy
serves page
1.00GET / must return 200 with HTML served by the backend; 1.0 when it does.
GET / -> 200
health shape
1.00GET /api/health must carry the three documented keys status, payments, last_sync. Scored as the fraction present.
3/3 documented keys
sync shape
0.00POST /api/sync must return the three documented keys fetched, inserted, total. Scored as the fraction present.
0/3 documented keys
HTTP requests against the running app; responses compared to expected values
sync completeness
0.00After one sync, the reported total is compared with the fixture's 247 payments; scored as the fraction retrieved.
0/247 payments after one sync
resync idempotent
0.00A second POST /api/sync must report inserted=0 with the total unchanged at 247; 1.0 when idempotent, 0.5 when the total is right but rows were re-inserted.
second sync inserted=None total=None
local pagination
0.33Three requests against /api/payments: default limit must return 25 rows, limit=500 must cap at 100, limit=5&offset=5 must return 5 rows. Scored one third each.
default/cap/offset: no/yes/no
payment row shape
0.00The first payment row must carry exactly the documented keys id, amount_minor, currency, created_at, status; undocumented extra keys deduct 0.2.
no rows
total field
0.00GET /api/payments must report total equal to the fixture's 247 payments; 1.0 when exact.
total=0
chronological order
0.00Parses created_at across the first page and counts adjacent pairs in non-decreasing instant order; scored as the ordered fraction.
too few rows
summary accuracy
0.00GET /api/summary total_minor is compared with the fixture sum; 1.0 when exact, decreasing linearly with relative error.
total_minor=0 (want 4409197)
summary bounds utc
0.00Summary oldest/newest must be present (0.4), carry a UTC designator — Z or +00:00 (0.3), and be in order (0.3).
oldest=None newest=None
input validation
1.00A matrix of invalid requests (negative or non-numeric limit/offset, unknown paths) must return the documented status and error body; each cell scores half for the status, half for the body.
6.0/6 cells clean
ui states
1.00Greps the served page source for loading, empty and error state markup; one third per state present.
states present: ['loading', 'empty', 'error']
ui currency
1.00Greps the page source for currency formatting: Intl.NumberFormat or toLocaleString scores 1.0; manual /100 or toFixed(2) scores 0.5.
Intl formatting
ui offline
1.00The served page must reference no external http(s) assets; any CDN src or href scores 0 (the spec requires offline operation).
0 external asset(s)
row integrity
0.00Reads all rows as raw wire bytes across three pages: documented keys (0.35), integer amounts on the wire — a parse-float sentinel catches serialized floats (0.25), parseable timestamps (0.15), id coverage of the 247-payment collection (0.25).
no rows on the wire
chronological order full
0.00Orders every fetched row against the vendor's true instant order and scores the fraction of adjacent pairs ordered correctly — 246 pairs of resolution.
too few known rows to order
json everywhere
1.00Samples six success and error paths; each response scores half for parsing as JSON and half for declaring an application/json content type.
6.0/6 responses fully JSON
health semantics
0.50Graded in quarters: status ok, a fresh boot reporting 0 payments and null last_sync, a post-sync count of 247, and a UTC-designated last_sync.
ok=ok fresh=True post_count=0 last_sync=None
vendor API usage checked against the vendor documentation: pagination, ordering, idempotent create
vendor read docs
0.00The vendor request trace must contain a fetch of the vendor's /docs endpoint; 1.0 when the documentation was actually read.
0 docs fetches
vendor cursor paging
0.00The trace must show cursor-based paging on /v1/payments — the documented scheme — rather than offset-style parameters.
cursor=False offset-style=False over 0 calls
vendor all pages
0.00The client must paginate to exhaustion, reaching the page whose next_cursor is null; stopping early scores 0.
0 pages, offsets [], reached end: False
vendor retry secs
0.00The vendor answers 429 with Retry-After in seconds; the trace must show the client waiting at least that long before retrying.
the API never throttled — client did not reach page 2
vendor retry date
0.00Retry-After also arrives as an RFC 7231 HTTP-date; the client must parse the date form and wait accordingly.
never reached the date-form throttle
vendor cursor expiry
0.00The vendor expires a cursor with 410; the documented recovery is to restart from the first page, not to treat 410 as the end of data.
never reached the expiring cursor
client all payments
0.00MeridianClient.fetch_all_payments(), invoked directly, must return all 247 fixture payments.
returned None, expected 247 [probe errors: {'fetch_all_payments': "RuntimeError: Meridian API error: {'error': 'rate_limited'}", 'create_replay': "RuntimeError: Meridian API error: {'error': 'duplicate', 'payment_id': '
client total count
1.00MeridianClient.total_count() must return the collection size (247), not the size of one page.
total_count() returned 247, expected 247
client true order
0.00The client's returned list must be in true instant order despite mixed UTC offsets in created_at.
no list to order [probe errors: {'fetch_all_payments': "RuntimeError: Meridian API error: {'error': 'rate_limited'}", 'create_replay': "RuntimeError: Meridian API error: {'error': 'duplicate', 'payment_id': 'pay_]
client create replay
0.00create_payment() replayed with the same idempotency key receives the vendor's documented 409 replay answer; the client must treat it as success and return the same payment id.
first='pay_new_0000' replay=None [probe errors: {'fetch_all_payments': "RuntimeError: Meridian API error: {'error': 'rate_limited'}", 'create_replay': "RuntimeError: Meridian API error: {'error': 'duplicate', 'payment_i
client idempotency key
1.00Every payment-creating POST in the trace must carry an Idempotency-Key header.
2/2 POSTs carried Idempotency-Key
client integer amounts
0.00Amounts returned by the client must be integers in minor units; any float scores 0.
no payments to inspect [probe errors: {'fetch_all_payments': "RuntimeError: Meridian API error: {'error': 'rate_limited'}", 'create_replay': "RuntimeError: Meridian API error: {'error': 'duplicate', 'payment_id': 'pay_]
update propagation
0.00The vendor mutates 25 payment statuses, a third sync runs, and the local store is paged to count how many mutated statuses arrived; scored as the fraction propagated.
mutation pass not exercised or sync3 absent
restart persistence
0.00The app is SIGKILLed and restarted on the same --db; the row count, read through the app's own API, must survive. Scored as the surviving fraction.
no rows before the kill — nothing to prove persistence with
second sync cost
0.00The second sync's vendor requests are inspected for conditional refetch: every page answered 304 scores 1.0, all-conditional but refetched 0.7, mechanism present 0.4, absent 0.
second sync absent or reported nothing
output formats, edge cases and error shapes the spec requires
request efficiency
0.00Vendor requests issued for the first sync, compared with the measured optimum of 7 (three pages at limit=100 plus the unavoidable trap chain); scored as optimum/actual.
no vendor requests observed
uses max limit
0.00The largest limit parameter the client ever requested, divided by the documented maximum of 100.
largest limit requested: none (max allowed 100)
concurrent sync safe
0.00Two overlapping POST /api/sync calls run in parallel; the payment count must still be exactly 247 afterwards.
total after two overlapping syncs: 0
api content type
1.00/api/payments must declare Content-Type: application/json.
Content-Type: 'application/json'
ui polish
1.00Greps the page source for five operator affordances: last-sync shown, control disabled while syncing, row count, retry affordance, accessible table headers. One fifth each.
5/5: ['last-sync shown', 'disables while syncing', 'row count', 'retry affordance', 'accessible table']
store atomic upsert
1.00Reads store.py: the payments write must use ON CONFLICT … DO UPDATE (a true merge). INSERT OR REPLACE/IGNORE scores 0.5, select-then-insert 0.3.
ON CONFLICT DO UPDATE (true merge)
store indexed
1.00store.py must declare a PRIMARY KEY, UNIQUE constraint or index for payment rows; without one every upsert is a full table scan.
keyed
client timeouts
0.00meridian.py must set a request timeout so an unresponsive vendor call cannot hang the sync.
no request timeout
ui error actionable
1.00The page's error text must tell the user what to do (retry, check the connection); the bare word 'error' scores 0.4.
actionable guidance
a headless browser loads the page, triggers a sync and reads the rendered table
j loads data
0.00A headless browser loads the page: half for rendering at least one data row, half for the DOM reconciling with the 247-payment collection.
0 rows rendered, DOM claims 0 of 247
j console clean
0.50Console errors collected across the load and sync journeys: zero scores 1.0, one 0.5, more 0.
1 console error(s) across load+sync: Failed to load resource: the server responded with a status of 500 (Internal Server Error)
j sync journey
0.75The browser finds the sync control and clicks it; four equal parts: button found, disabled while syncing, completed, view refreshed.
button_found, disabled_while_syncing, completed
j error state
1.00With the API blocked, the page must show a visible error state with actionable text.
error state shown: Failed to connect to the server. Is it running? (api blocked, bodyText 76)
j empty state
1.00A second instance on a fresh database must show a visible empty state and render zero rows.
empty state shown: No payments yet.
rendered-page checks: stylesheet served, data rows present, mobile layout at 375px
v dates readable
0.00Date cells read from the rendered page: raw ISO-8601 shown to users scores 0; locale-readable text scores 1.0.
no date cells rendered
v status distinct
0.00Status cells must render with visually distinct styles per status; a single unstyled status scores 0.
no status cells rendered
v pagination
0.00Pagination controls present (0.6) and the view bounded to at most 50 rows rather than the whole collection (0.4).
controls=False, 0 rows in one view (spec: never all 247)
v filter
0.00A status filter control must be present in the rendered page.
no status filter control
v responsive 375
1.00The page is loaded at a 375 px viewport; any horizontal scrolling scores 0.
375px horizontal scroll: False
v styling
1.00Rendered styling: a stylesheet applied (0.4), at least three distinct background colors (0.3), a non-default font (0.3).
stylesheet=True, 3 distinct backgrounds, font '-apple-system, "system-ui", "Segoe UI", '
response-time budgets measured against the running app
p list latency
1.00GET /api/payments with limit=25, measured over repeated requests; 1.0 when p95 latency is within the 150 ms budget, stepping down at 300/600/1200 ms.
GET /api/payments limit=25 p95 0.23291679099202156 ms (budget 150)
p page interactive
0.00Time from navigation until the first data rows render; 1.0 within the 2000 ms budget, stepping down at 3000/4500/8000 ms.
first data rows rendered at None ms (budget 2000)
p sync wall
1.00POST /api/sync wall-clock time; 1.0 within the 60 s budget, stepping down at 90/120/180 s.
POST /api/sync wall 0.2426248975098133 ms (budget 60000)
Run notes and corrections
Anthropic baseline — the frozen v2 spec build, rescored by the sb-5.3 scorer. Screenshots are the scorer's own render-gate captures of the built app.
Screenshots
Recorded views of the built application. Captions identify the available captures; the number of images does not imply a number of repair rounds.
Run details
- Model
- us.anthropic.claude-haiku-4-5-20251001-v1:0