It started with a dashboard that wouldn't finish loading. All the built-in checks failed me — the deploy was clean, the data was in the tables — and the dashboard just sat on its spinner until the test timed out. Hours of dead ends later I opened the browser's JS console by hand: epochMilliseconds must be a number, you gave: null. It had been printing there the whole time, in the one place neither I nor the tests were looking.

The bug itself got a type fix (the date parameter's default became a required field, so the wiring site can't omit it again). The fix that mattered was to the harness: browser test failures now capture the JS console automatically. That was the first bolt-on of many: when every test drives the browser through ONE shared layer, every piece of detection you add to that layer upgrades every test — including the ones written months ago.

Browser tests have a deserved reputation as the worst tests in a suite: selector soup and flake nobody can reproduce. The standard prescription is the Page Object pattern, and it's fine as far as it goes — it organizes the selectors. However! organizing selectors is the smaller half of the win. The bigger half is that a single choke-point between your tests and the browser gives you a PLACE to put issue detection. Nobody sold me on that half, I backed into it, so let me sell you.

The codebase here is recon-gen, my financial-reconciliation validation platform — dashboards full of filter pickers, sliders, KPIs and drill-through tables, driven under test by Playwright (the browser-automation library) against real WebKit (Safari's browser engine). The dashboards originally rendered two ways: Amazon QuickSight embeds and a self-hosted renderer.

Tests speak verbs

End-to-end (e2e) test bodies in this repo don't touch Playwright. They speak a DashboardDriver protocol (a typed interface) whose methods are verbs — things an operator does: open, pick_filter, table_rows, set_date_range, drill_from_first_row. Every verb returns plain Python, never a Page or a Locator, and every write verb blocks until the affected visuals (the dashboard's individual charts, tables and KPI tiles) have re-fetched — the driver calls this settling — so the read on the next line sees post-pick state without the test author ever writing a wait.

It didn't start that way. The pre-port tests each carried five browser-helper imports, an embed-URL fixture, a with webkit_page(...) as page: block, a goto, a settle wait and then the actual assertion buried at the bottom. The port turned that into (lightly trimmed; the arguments are pytest fixtures):

def test_exec_dashboard_lists_all_sheet_tabs(
    qs_driver, exec_dashboard_id, exec_app,
) -> None:
    """Every sheet the dashboard definition declares shows up
    as a tab on the deployed dashboard."""
    qs_driver.open(exec_dashboard_id)
    expected = {s.name for s in exec_app.analysis.sheets}
    tabs = set(qs_driver.sheet_names())
    missing = expected - tabs
    assert not missing, f"Missing sheet tabs: {missing}. Found: {sorted(tabs)}"

All 20 legacy files got ported in one day. The accounting from git diff --stat: test bodies lost a net 846 lines while the shared driver layer grew 384 — and every test written since gets those 384 lines for free. Today the tests make ~470 verb calls across a vocabulary of ~65 verbs, and the vocabulary is rich enough that "the test asks the wrong question" becomes a one-line fix (one flaky assertion turned out to be reading a virtualized dropdown that only mounts ~12 options; the fix was swapping filter_options for typeahead_filter — type to narrow, same as a real operator).

The original sales pitch for the layer was multi-renderer: one test body, parametrized [qs, app2], ran against BOTH a deployed QuickSight embed (the qs leg) and the self-hosted renderer (the app2 leg); the app2 leg found three real renderer gaps its first run. Then QuickSight got ripped out of the project entirely (why is its own post) and the parametrize collapsed to one renderer — and the driver layer stayed anyway. The parametrization was never the real reason. The real reason is that a test that reads as what the operator DID is a test you can still reason about six months later.

The whole shape of the thing:

direction: down

tests: e2e tests — ~100 bodies, zero Playwright imports
driver: DashboardDriver — the verb layer {
  verbs: open · pick_filter · table_rows · drill_from_first_row · …
  detection: SQL-exception scan · entity scan · expected-call screams
  instrumentation: console sink · network log · always-on trace
}
browser: Playwright → WebKit → the app
artifacts: "on failure: screenshot.png · dom.html · console.txt · network.txt · db_counts.txt · trace.zip"

tests -> driver: speaks verbs, gets plain Python back
driver -> browser: owns every selector, wait and quirk
driver -> artifacts: dumps the bundle on any failure
tests -> browser: "✗ import playwright — the lint fails the build" {style.stroke-dash: 3}

What lives under one verb

A sampling, all of it triage-earned and all of it QuickSight-era (the renderer died; burying its quirks in verbs is the part that transfers):

• QuickSight charts render to <canvas> — there are no DOM bars to click. Selecting a bar is: focus the visual's whitespace, Tab five times, Enter to highlight, arrow keys to your category, Enter again. (Pie charts don't expose the keyboard path at all, which is how bar charts became the automatable default.)
• QuickSight's date picker commits on focus-loss, not Enter. fill + Enter leaves the input LOOKING correct while the dataset SQL silently keeps its wide-open 1990/2099 defaults — latent for three months, because every screenshot showed the picked date. The table dropped from 5,551 rows to 42 the moment the verb learned to press Tab.
• Tables virtualize: a 119-row table mounts ~10–37 rows in the DOM. Reading one honestly means scroll-and-accumulate with stability counters and a bounded step budget.

Each of these was diagnosed once, fixed in the verb, and every test touching a chart, a date or a table healed in the same commit. The alternative is that knowledge copy-pasted across test files, each copy drifting on its own schedule.

Every failure arrives pre-triaged

The driver's page factory arms instrumentation at page creation: a console + pageerror sink, a network log, Playwright tracing. On a passing test it all evaporates. On a failure it lands as a bundle next to the test id — the core six: screenshot.png, dom.html (what the selectors actually saw — pairs with the pixels), console.txt, network.txt, db_counts.txt (per-table row counts, because a blank visual is two different failure modes — backend-empty vs frontend-broken — and you want to know which before any DOM archaeology) and trace.zip for time-travel replay. Disk is cheap compared to time; the harness keeps the last 20 runs and prunes.

One tuning detail that earned its keep: the network log records every non-2xx response PLUS every visual-data response regardless of status, because a 200 with zero rows looks identical to "no request fired" in a filtered log — and that distinction is exactly the bug class you're chasing when a filter pick goes wrong.

To be clear, this layer is forensic — nothing fails a passing test over a console message. Its job is making the failure that already happened diagnosable without a re-run. A real one from this week — the two lines of a failing test's network.txt where the log flips:

200 GET .../visuals/227d12b1.../data?...&param_pInvMoneyTrailMinAmount=0
500 GET .../visuals/227d12b1.../data?...&param_pInvMoneyTrailMinAmount=270404.48

That's the whole diagnosis in two lines: the filter slider committed a fractional dollar amount into an integer parameter, and of the SQL engines the suite runs against, DuckDB silently coerces it where PostgreSQL refuses. The fix landed 23 minutes after the failing run, without a redeploy or a debugger session.

Verbs that scream

The hard-fail detection lives in the verbs themselves, and it accumulated the same way — each layer started as one confusing failure:

• SQL-exception scanning. A test failure read "Transactions went empty after picking" — but the screenshot showed every visual rendering QuickSight's "Your database generated a SQL exception" widget. The driver had counted zero rows and called it a legitimate filter-narrowed-to-empty. Wrong diagnosis, and a page in that state makes every other assertion on it useless — so now every page-changing verb scans for the SQL-exception state first and fails with the actual database error. The capture even fires BEFORE the raise, because by the time pytest teardown runs, QuickSight has retried and repainted the evidence away.
• Escape-bug scanning. I spotted literal 's in page chrome — titles, nav, banners. The fix wasn't a find-and-patch sweep: the driver now scans the DOM of every page any browser test loads for double-encoded entities. The next day it missed one hiding inside a code span; tightened within 24 hours.
• Expected-call verification. When a verb's mutation is supposed to cause a server call, check it fired; if not, scream. The driver records the requests around each mutation and names the specific gap instead of decaying into an opaque 30-second timeout:

direction: down

pick: "pick_filter(\"Rail\", [\"ACH\"])"
q1: search request fired? {shape: diamond}
q2: value in the returned options? {shape: diamond}
q3: visual refetch fired? {shape: diamond}
ok: settle — return plain Python
s1: "SCREAM: front-end wiring gap (the expected call never happened)"
s2: "SCREAM: backend data gap (request fired, value never came back)"
s3: "SCREAM: the mutation fired NO refetch"

pick -> q1
q1 -> s1: no
q1 -> q2: yes
q2 -> s2: no
q2 -> q3: "yes → setValue"
q3 -> s3: no
q3 -> ok: yes

The week this landed it cracked a bug I'd been misdiagnosing off an incomplete network log: the test's anchor account was declared in the data spec but never materialized in the seed (the synthetic dataset the suite generates for every run). Then it went on to disprove a standing "picker race" excuse and expose the seed bug hiding under it.

None of the tests changed for any of this; the detection is retroactive to every test that already calls the verb.

Lints seal the boundary

None of the above survives without enforcement. A driver layer is a convention, and conventions rot — the first time someone is in a hurry (me, or these days the AI drafting tests under my direction), raw Playwright creeps back into a test body and the layer starts leaking. So the boundary is a lint, not a code-review comment: an abstract-syntax-tree (AST) check — a unit test that parses the test suite's own source — fails the build on import playwright anywhere in the e2e tree outside the driver directory. It shipped the same day as the driver protocol, and its failure message teaches the fix — tests talk verbs, Playwright stays sealed behind the driver. (There IS a documented escape hatch — driver.page for assertions about the raw request/response shape that don't translate to verbs, plus three per-line suppressions each carrying a written reason.)

Two details I'd repeat in any codebase:

• The ratchet. The lint shipped while 20 files still violated it, with the offenders listed in a frozenset the lint skips. Porting a file removes its name, so the set can only shrink — it's been literally frozenset() — empty — since the port finished, the migration receipt compiled into the enforcement.
• The lints get tests too. A lint that's supposed to find zero hits is indistinguishable from a lint that's broken. So the convention is that a lint meant to land at zero hits ships with a planted-violation fixture plus a smoke test asserting the exact hit count — lint death is loud instead of silent.

The lint vocabulary grew the same way the verbs did, one incident at a time:

• A test inlined a dashboard title that the driver takes as an argument; production renamed the title and the test broke silently for a week. The constants lint (which already flagged production constants inlined into test bodies) now checks verb arguments too — extending it immediately surfaced ~187 more inlined values.
• Sleeps flake in both directions (too short is a race, too long is slow CI), and waiting is the driver's job anyway — so time.sleep is banned across the e2e tree. Exactly two suppressions exist in the corpus, both 50ms server-startup polls with a written why.
• After two bugs traced to tests poking hidden form inputs, an AST check screams if the string "hidden" appears in any e2e test at all — driving hidden DOM is the driver's job; tests drive what the user can see. (Same principle as the drivers locating by visible labels and ARIA roles, never CSS classes: the test should break when the user's experience breaks, not when the styling shifts.)

All of it — two dozen lints — runs as one pytest test in ~13 seconds, no browser, no database.

The bill

The layer isn't free.

It's real code — ~4,900 lines of drivers plus a 2,700-line helper module — with real bugs of its own. The settle logic alone has been wrong four separate times:

• disk-cache hits don't fire Playwright's response events, so a cached re-fetch looked like no re-fetch at all;
• a stale queued refresh cleared the loading skeleton while a fresher request was still in flight;
• a two-input date write raced its own settle snapshot;
• a row read raced the repaint — the failure screenshot showed 4 rows on screen while the driver had read 0. The instrumentation convicted the driver.

The failure capture itself was silently dead for a stretch: pytest never re-throws a test failure into a yield-fixture (the setup/teardown hook the capture lived in), so the obvious try/except design never fired — which is why the capture wiring now has its own regression test. And the instrumentation can BE the bug: always-on trace filmstrips leaked GPU surfaces on macOS until the window compositor crashed out from under the test run.

If you have five browser tests, none of this is worth it — write them raw and move on. At this suite's scale (the browser layer alone collects 114 tests and finishes in ~2.5 minutes), the verbs are why one person can maintain it and a failing run costs minutes instead of an evening.

Where to start

Not with 40 verbs. This order worked for me:

1. Bug the page factory first — console + pageerror sink, a network log, screenshot and DOM dump on failure. It converts your next mystery failure into a text file you can read.
2. Grow verbs out of repetition, not up-front design. Every one of the ~65 started as either duplicated plumbing or a triage discovery that needed a home. A vocabulary designed in advance would have invented the wrong words.
3. Write the leak-lint the same day you write the driver, and plant a violation so you know the lint itself is alive. It's ~50 lines of AST walk.

Getting Playwright to the point where test cases are easy to write was a grueling journey — grueling enough that when another codebase of mine needs browser tests now, I don't rebuild any of this, I point it at this repo and steal the driver.

Start with the console capture. Worst case you're out an afternoon; best case you never again learn about a JavaScript error hours late, from a spinner that won't stop.

Cover Image from Shakespeare Memorial Theatre by Alan Murray-Rust CC BY-SA 2.0 via Wikimedia Commons