Benchmark run

3-node fleet — full fix-chain run0.9300 on sb-5.3

qwen3.6-27b-fable-fusion-711-uncensored-heretic-nm-dau-neo-max-mtp · 3 nodes · scorer sb-5.3

mighty-crane-54f23-node local fleetAug 18, 2026
Overall
0.9300
Hard
0.83
Excellent
yes
Scorer
sb-5.3
Wall clock
190 min
Nodes
3

Tier breakdown

A · structure & runtime1.00
B · behaviour1.00
C · vendor contract1.00
D · finesse0.94

Scoring detail

How this score was built

Per-check scorer output, as posted by the app.

ComponentWeightEarnedPoints
Core (tiers A–D)60%0.9880.593
A · Structure25% of core1.000.150
B · Behaviour30% of core1.000.180
C · Vendor contract25% of core1.000.150
D · Finesse20% of core0.940.112
Journey (J)15%0.7500.112
Visual (V)10%0.9170.092
Performance (P)5%1.0000.050
Hard blocks10%0.8330.083
Total0.9300
AStructurefiles, modules and interfaces named by the spec exist; the server starts and binds

modules present

1.00

Checks that the five files the spec names by path exist: meridian.py, store.py, api.py, __main__.py, web/index.html. Scored as the fraction present.

5/5 named files

interfaces declared

1.00

Parses meridian.py and store.py and checks the eight named methods are declared on MeridianClient (fetch_all_payments, total_count, create_payment) and Store (upsert_many, all_payments, count, last_sync, set_last_sync). Scored as the fraction present.

8/8 methods

server runs

1.00

Starts python -m vendorsync on a free port and polls GET /api/health for up to 25 s; 1.0 when the server binds and answers 200.

healthy

serves page

1.00

GET / must return 200 with HTML served by the backend; 1.0 when it does.

GET / -> 200

health shape

1.00

GET /api/health must carry the three documented keys status, payments, last_sync. Scored as the fraction present.

3/3 documented keys

sync shape

1.00

POST /api/sync must return the three documented keys fetched, inserted, total. Scored as the fraction present.

3/3 documented keys

BBehaviourHTTP requests against the running app; responses compared to expected values

sync completeness

1.00

After one sync, the reported total is compared with the fixture's 247 payments; scored as the fraction retrieved.

247/247 payments after one sync

resync idempotent

1.00

A second POST /api/sync must report inserted=0 with the total unchanged at 247; 1.0 when idempotent, 0.5 when the total is right but rows were re-inserted.

second sync inserted=0 total=247

local pagination

1.00

Three requests against /api/payments: default limit must return 25 rows, limit=500 must cap at 100, limit=5&offset=5 must return 5 rows. Scored one third each.

default/cap/offset: yes/yes/yes

payment row shape

1.00

The first payment row must carry exactly the documented keys id, amount_minor, currency, created_at, status; undocumented extra keys deduct 0.2.

['amount_minor', 'created_at', 'currency', 'id', 'status']

total field

1.00

GET /api/payments must report total equal to the fixture's 247 payments; 1.0 when exact.

total=247

chronological order

1.00

Parses created_at across the first page and counts adjacent pairs in non-decreasing instant order; scored as the ordered fraction.

24/24 adjacent pairs ordered

summary accuracy

1.00

GET /api/summary total_minor is compared with the fixture sum; 1.0 when exact, decreasing linearly with relative error.

total_minor=4409197 (want 4409197)

summary bounds utc

1.00

Summary oldest/newest must be present (0.4), carry a UTC designator — Z or +00:00 (0.3), and be in order (0.3).

oldest=2026-03-01T12:00:00Z newest=2026-03-07T08:36:00Z

input validation

1.00

A matrix of invalid requests (negative or non-numeric limit/offset, unknown paths) must return the documented status and error body; each cell scores half for the status, half for the body.

6.0/6 cells clean

ui states

1.00

Greps the served page source for loading, empty and error state markup; one third per state present.

states present: ['loading', 'empty', 'error']

ui currency

1.00

Greps the page source for currency formatting: Intl.NumberFormat or toLocaleString scores 1.0; manual /100 or toFixed(2) scores 0.5.

Intl formatting

ui offline

1.00

The served page must reference no external http(s) assets; any CDN src or href scores 0 (the spec requires offline operation).

0 external asset(s)

row integrity

1.00

Reads all rows as raw wire bytes across three pages: documented keys (0.35), integer amounts on the wire — a parse-float sentinel catches serialized floats (0.25), parseable timestamps (0.15), id coverage of the 247-payment collection (0.25).

247 rows: keys 247/247, int-amounts 247/247, times 247/247, ids 247/247

chronological order full

1.00

Orders every fetched row against the vendor's true instant order and scores the fraction of adjacent pairs ordered correctly — 246 pairs of resolution.

246/246 adjacent pairs in true instant order

json everywhere

1.00

Samples six success and error paths; each response scores half for parsing as JSON and half for declaring an application/json content type.

6.0/6 responses fully JSON

health semantics

1.00

Graded in quarters: status ok, a fresh boot reporting 0 payments and null last_sync, a post-sync count of 247, and a UTC-designated last_sync.

ok=ok fresh=True post_count=247 last_sync=2026-08-18T13:46:29Z

CVendor contractvendor API usage checked against the vendor documentation: pagination, ordering, idempotent create

vendor read docs

1.00

The vendor request trace must contain a fetch of the vendor's /docs endpoint; 1.0 when the documentation was actually read.

9 docs fetches

vendor cursor paging

1.00

The trace must show cursor-based paging on /v1/payments — the documented scheme — rather than offset-style parameters.

cursor=True offset-style=False over 25 calls

vendor all pages

1.00

The client must paginate to exhaustion, reaching the page whose next_cursor is null; stopping early scores 0.

22 pages, offsets [0, 100, 200], reached end: True

vendor retry secs

1.00

The vendor answers 429 with Retry-After in seconds; the trace must show the client waiting at least that long before retrying.

waited 2.01s, Retry-After said 2s

vendor retry date

1.00

Retry-After also arrives as an RFC 7231 HTTP-date; the client must parse the date form and wait accordingly.

retried +0.00s relative to the stated moment

vendor cursor expiry

1.00

The vendor expires a cursor with 410; the documented recovery is to restart from the first page, not to treat 410 as the end of data.

restarted from page 0: True; re-sent the dead cursor first: False

client all payments

1.00

MeridianClient.fetch_all_payments(), invoked directly, must return all 247 fixture payments.

returned 247, expected 247

client total count

1.00

MeridianClient.total_count() must return the collection size (247), not the size of one page.

total_count() returned 247, expected 247

client true order

1.00

The client's returned list must be in true instant order despite mixed UTC offsets in created_at.

chronological

client create replay

1.00

create_payment() replayed with the same idempotency key receives the vendor's documented 409 replay answer; the client must treat it as success and return the same payment id.

first='pay_new_0000' replay='pay_new_0000'

client idempotency key

1.00

Every payment-creating POST in the trace must carry an Idempotency-Key header.

2/2 POSTs carried Idempotency-Key

client integer amounts

1.00

Amounts returned by the client must be integers in minor units; any float scores 0.

0 non-integer amounts of 247

update propagation

1.00

The vendor mutates 25 payment statuses, a third sync runs, and the local store is paged to count how many mutated statuses arrived; scored as the fraction propagated.

25/25 mutated statuses visible locally after sync3

restart persistence

1.00

The app is SIGKILLed and restarted on the same --db; the row count, read through the app's own API, must survive. Scored as the surviving fraction.

247/247 rows survived kill+reboot on the same db

second sync cost

0.00

The second sync's vendor requests are inspected for conditional refetch: every page answered 304 scores 1.0, all-conditional but refetched 0.7, mechanism present 0.4, absent 0.

0/9 conditional, 0/9 answered 304

DFinesseoutput formats, edge cases and error shapes the spec requires

request efficiency

1.00

Vendor requests issued for the first sync, compared with the measured optimum of 7 (three pages at limit=100 plus the unavoidable trap chain); scored as optimum/actual.

7 vendor requests for the first sync (optimum 7)

uses max limit

1.00

The largest limit parameter the client ever requested, divided by the documented maximum of 100.

largest limit requested: 100 (max allowed 100)

concurrent sync safe

1.00

Two overlapping POST /api/sync calls run in parallel; the payment count must still be exactly 247 afterwards.

total after two overlapping syncs: 247

api content type

1.00

/api/payments must declare Content-Type: application/json.

Content-Type: 'application/json'

ui polish

1.00

Greps the page source for five operator affordances: last-sync shown, control disabled while syncing, row count, retry affordance, accessible table headers. One fifth each.

5/5: ['last-sync shown', 'disables while syncing', 'row count', 'retry affordance', 'accessible table']

store atomic upsert

0.50

Reads store.py: the payments write must use ON CONFLICT … DO UPDATE (a true merge). INSERT OR REPLACE/IGNORE scores 0.5, select-then-insert 0.3.

INSERT OR REPLACE/IGNORE — writes but does not merge

store indexed

1.00

store.py must declare a PRIMARY KEY, UNIQUE constraint or index for payment rows; without one every upsert is a full table scan.

keyed

client timeouts

1.00

meridian.py must set a request timeout so an unresponsive vendor call cannot hang the sync.

timeout set

ui error actionable

1.00

The page's error text must tell the user what to do (retry, check the connection); the bare word 'error' scores 0.4.

actionable guidance

JJourneya headless browser loads the page, triggers a sync and reads the rendered table

j loads data

1.00

A headless browser loads the page: half for rendering at least one data row, half for the DOM reconciling with the 247-payment collection.

25 rows rendered, DOM claims 247 of 247

j console clean

1.00

Console errors collected across the load and sync journeys: zero scores 1.0, one 0.5, more 0.

0 console error(s) across load+sync

j sync journey

0.75

The browser finds the sync control and clicks it; four equal parts: button found, disabled while syncing, completed, view refreshed.

button_found, disabled_while_syncing, completed

j error state

0.00

With the API blocked, the page must show a visible error state with actionable text.

error state NOT shown (api blocked, bodyText 154)

j empty state

1.00

A second instance on a fresh database must show a visible empty state and render zero rows.

empty state shown: No payments yet — click Sync now to fetch from Meridian.

VVisualrendered-page checks: stylesheet served, data rows present, mobile layout at 375px

v dates readable

1.00

Date cells read from the rendered page: raw ISO-8601 shown to users scores 0; locale-readable text scores 1.0.

rendered dates like 01 Mar 2026, 14:00

v status distinct

0.50

Status cells must render with visually distinct styles per status; a single unstyled status scores 0.

only 'Settled' rendered on page 1; badge-styled

v pagination

1.00

Pagination controls present (0.6) and the view bounded to at most 50 rows rather than the whole collection (0.4).

controls=True, 25 rows in one view (spec: never all 247)

v filter

1.00

A status filter control must be present in the rendered page.

status filter control present

v responsive 375

1.00

The page is loaded at a 375 px viewport; any horizontal scrolling scores 0.

375px horizontal scroll: False

v styling

1.00

Rendered styling: a stylesheet applied (0.4), at least three distinct background colors (0.3), a non-default font (0.3).

stylesheet=True, 3 distinct backgrounds, font '-apple-system, "system-ui", "Segoe UI", '

PPerformanceresponse-time budgets measured against the running app

p list latency

1.00

GET /api/payments with limit=25, measured over repeated requests; 1.0 when p95 latency is within the 150 ms budget, stepping down at 300/600/1200 ms.

GET /api/payments limit=25 p95 0.47166598960757256 ms (budget 150)

p page interactive

1.00

Time from navigation until the first data rows render; 1.0 within the 2000 ms budget, stepping down at 3000/4500/8000 ms.

first data rows rendered at 18 ms (budget 2000)

p sync wall

1.00

POST /api/sync wall-clock time; 1.0 within the 60 s budget, stepping down at 90/120/180 s.

POST /api/sync wall 3.506041830405593 ms (budget 60000)

Findings that held

Raised by the verify gate, still open when the run ended.

1

`pytest -q` failed — the generated tests exercise runtime paths that `--help`/`--collect-only` never invoke: client = MeridianClient("http ... [middle elided — head + tail shown] ... ") > self.assertEqual(captured_req[0].headers["Idempotency-Key"], "key-abc") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E KeyError: 'Idempotency-Key' tests/test_merid

2

POST /api/sync is not CHEAP on a repeat run — the second sync re-fetched 247 row(s) it already had. FIX: make the client send If-None-Match per page — ... [middle elided — head + tail shown] ... lay THAT page's ETag on the matching request; treat 304 as 'page unchanged, keep local rows'. One ETag replayed on every page never matches and re-fetches everything. Rows are correct, so this is not a d

3

GET /api/summary answered while the app was EMPTY and returns NOTHING AT ALL once it holds rows — same process, same endpoint, one sync in between, and its sibling endpoints still answer. That pair rules out both a slow endpoint and a dead server, and points at the code path that reads STORED rows: an

4

the page renders but the browser console carries 4 error(s) in normal use (first: Failed to load resource: net::ERR_EMPTY_RESPONSE) — fix the JS errors; users hit them as broken interactions.

Repair progression

R0·14R1·4

One chip per verify round: the number of open findings at that round. The run ends when a round reports zero findings or the round budget is exhausted.

Screenshots

Captured by the render gate during the run's repair rounds. The first capture is the initial render; the remainder are from the final epoch.

3-node fleet — full fix-chain run on the sb-5.3 agentic benchmark — First render — before repairs
1First render — before repairs
3-node fleet — full fix-chain run on the sb-5.3 agentic benchmark — Final render
2Final render
3-node fleet — full fix-chain run on the sb-5.3 agentic benchmark — Mobile · 375px
3Mobile · 375px
3-node fleet — full fix-chain run on the sb-5.3 agentic benchmark — After sync
4After sync

Run details

Model
qwen3.6-27b-fable-fusion-711-uncensored-heretic-nm-dau-neo-max-mtp
Engine events
801
Repair rounds
1
Started
Aug 18, 2026, 10:36 AM
Finished
Aug 18, 2026, 01:46 PM

Fleet nodes

NodeModel
gabeegabee-qwen3.6-27b-fable-fusion-711-uncensored-heretic-nm-dau-neo-max-mtp
mihaimihai-qwen3.6-27b-fable-fusion-711-uncensored-heretic-nm-dau-neo-max-mtp
workhorseworkhorse-qwen3.6-27b-fable-fusion-711-uncensored-heretic-nm-dau-neo-max-mtp