There's a comment in a Python file somewhere that reads # type: ignore[no-untyped-def]. It looked like type protection. It was written in mypy's rule-code syntax. The project uses pyright. Pyright silently ignores mypy's codes. So the comment did nothing. The author aka me believed they had coverage; the gate was a no-op.

Then a typo'd attribute access — cfg.as_of_frame(...) against an object that had no such method — sailed straight through. Pyright catches that in three seconds. Instead it burned a full 22-minute CI run before anyone saw it.

This is the challenge of developing with AI agents, they have a lot of intelligence, little wisdom and the memory of a goldfish. In isolation that check was fine, however in THIS codebase that was a landmine we stepped in.

The challenge here is not unique to an agentic AI project, EVERY code base that grows large enough will exhibit the same problems just like an organization growing larger. At a certain point you (or a context window) can't keep it in your head and just like an organization has processes to standardize and protect itself, a codebase needs the same.

Even worse is the process or check that claims to work and doesn't, that poisons everything and wrecks your trust and ability to reason about the system.

Background

AI writes a large share of code now, including a large share of mine. I've built recon-gen, a financial-validation platform: roughly 120k lines of pyright-strict core, a test suite larger than the product itself (4,990 test functions as of this writing), held at zero type errors across 240+ tagged releases. All of the code was written AI-assisted.

The line count AND the test count is not the achievement. Any agent can emit plausible code by the thousand-line like a modern day Charles Dickens. The achievement is the control system around it: the rules and types and gates that make "the tests pass" actually mean "the code is correct," not "the agent found a way to make the failures go away."

To be clear it is a constant fight with an agent to not delete parts of the system or failing tests. It has NO intrinsic motivation, it's YOUR system not its. So when you say make it work, it sees a failure and needs pressure to not take the shortcut AND needs to know if its fix is going to cause problems in other areas (both spatially in other files and temporally in the form of tech debt).

Here's how I've approached this problem: Remove the need to remember things as much as possible.

Invariants in types, not in tests

The quickest place to catch a mistake is in the types, before a test ever runs. Here's the dumbest expensive bug in the platform: a DATETIME field bound to a string parameter. Untyped, that produces a dashboard that loads fine and shows zero rows. The mistake is in the wiring, the symptom is an empty screen three layers away. You debug the screen for an hour before you find the mismatch.

So the dataset-contract layer encodes column shapes as a typed enum that the checker treats as distinct, incompatible types (nominal subtyping — a DATETIME column won't pass where a string is expected, even though both are "just data"). A mis-wired binding fails at code-gen time AT the problematic line. The zero-row dashboard is now impossible to construct.

Side Note: Types bring meaning with them, convention hides it.

Any time the answer to "where does this go" or "is this range inclusive" lives in a docstring, that meaning gets lost at the next place someone calls the code. Take a bare DateInterval(start, end) — is that inclusive or exclusive of the end date? Nobody knows (and this example caused some subtle bugs), so the bare constructor doesn't exist. Only DateInterval.closed(s, e) and .trailing_days_ending_yesterday(...) do, and an AST lint (a check that reads the code's structure, not its types) rejects the bare DateInterval(...) call anywhere outside the type's own file.

I turned that same rule on the test suite itself, since tests are code too.

The custom linter catches typing smells pyright can't express: a bare str parameter that should be a named type, an explicit Any escape hatch, an env-var bypass. Even the test-tier labels are typed, so a typo'd tier("appp2") is an error while you're still typing it. Convention smell is type smell, all the way down.

When the type checker stops, I write a lint

Pyright is strict, but a type system only reaches so far. "Use the named constructor." "Don't paste a production value straight into a test." "Don't seed a random number generator at the top of a file." Those are real invariants and none of them are types. The usual home for a rule like that is a CONTRIBUTING doc, which is to say nowhere. Nobody reads it, the agent definitely doesn't, and the convention erodes one reasonable-looking change at a time.

So when the type system runs out, I extend the checker instead. recon-gen has around two dozen custom AST lints, each one a convention made unbreakable.

A few pointed examples:

• no-naked-interval-ctor. DateInterval(a, b) type-checks fine, but is it inclusive? Trailing N days? Anchored to the end of a window? The answer lived in someone's head. So the bare constructor is banned outside its own file: you call .closed(...) or .trailing_days_ending_yesterday(...) or it doesn't pass. The lint IS the doc, except it's enforced.
• no-inline-production-constants / no-test-src-sql-duplication. An agent writing a test will happily inline assert sheet_name == "Daily Statement" or paste a 40-line SQL string. This works and the check will be valid in the positive case, and that's the insidious issue. Negative tests WILL drift in the face of change and you will again have that false sense of security that the test harness is working when it isn't. That drift isn't even local to a change diff so no reviewer will even see it on a pull request. The only defense is to prohibit those strings from even appearing in tests.
• test-module-nondeterminism. This one's a scar. A single line — FUZZ_SEED = secrets.randbits(32) at the top of a file — made parallel test workers disagree on which tests even existed and refuse to start, with a baffling intermittent error (the kind you waste an afternoon on because it doesn't fail the same way twice). The fix wasn't to delete the line, it was an AST check that bans that whole class of randomness-at-import everywhere.

And then the part that makes the whole thing trustworthy. The most important of these lints are ones I expect to NEVER fire (the codebase is already clean), and a check that never fires is indistinguishable from a check that's quietly broken. So those lints carry a planted-violation fixture: a deliberately broken file the lint MUST catch. If a refactor breaks the lint, it stops catching the planted file and the suite fails for that reason alone. Which is the opening of this post generalized: a checker you never check is a checker that's lying to you. So I check the checkers too.

Negative Validations MUST test something

Back to the type: ignore that wasn't. The fix wasn't "audit all the comments and hope I find the rest." The fix was structural, and it's the same answer as the lints above: verify the gate fires, don't just trust that the run passed.

Any check that's expected to find nothing in normal operation gets a planted-violation fixture, a deliberately broken input the gate MUST flag. If the gate stops flagging it the gate is broken, and the suite fails for that reason alone. The dangerous state isn't "the check found nothing." It's "the check found nothing because it's EMPTY, not because it's actually looking." Those two are identical from the outside until you plant something it has to catch.

The same idea scales up. The negative-data generator produces around 30 declared kinds of invariant violation (drift, overdraft, limit breach, stuck-pending, phantom rails) all derived from the shape of the institution's own data. Every gate that's supposed to catch one of those is fed one on purpose. A passing board now means the catchers caught, not that the catchers were asleep.

No silent defer

One night an agent marked a failing test as "expected to fail" to make the suite pass. I came back to a clean board and a problem buried under it.

last night's decision to just xfail something was deeply offensive to me, knowingly sweeping a problem under the rug is begging to blow up in one's face.

To be clear, marking a test "expected to fail" (xfail) has legitimate uses, but here it was used to hide a real failure. That became a hard policy: no xfail, no skip, no "fix it later" backlog item. The fix ships in the same commit that found the failure.

Outlawing it once wasn't enough. Muting a failing test is the single most persistent thing I fight, the agent reaches for it again and again under pressure, so the rule is enforced in three places at once: the agent's standing instructions, its long-term memory and the plan tool that tracks the work. Ban it in only one of those and the agent quietly routes around it through the others. Defense in depth, against my own assistant, is the only thing that holds.

The payoff was immediate. The policy forced me to dig into three failures that had been waved off as "pre-existing." Every one was a real bug: a group of parallel tests that was never actually running, a freshness check that returned the wrong answer on empty input, a date-format scanner gotcha. All three would have rotted indefinitely under an xfail. The waiver isn't a deferral, it's a decision to never look again... until it grows and bites you!

Strictness is a correctness gift

I swapped the storage engine to DuckDB. It immediately crashed with a conflicting-lock error in a spot where the previous engine had "worked" just seemingly fine.

The previous engine, Sqlite's, file-level locking had been quietly serializing concurrent writers behind my back. It wasn't handling the concurrency, it was hiding it. Under that cover parallel test workers had been stomping each other's data the whole time and nothing said so. The loose engine hid the bug, the strict one made it loud.

the duckdb locks just saved us, we really should isolate things running in parallel unless they are all reading.

DuckDB's strict one-writer-per-file rule is a correctness gift, not an obstacle. The right fix was to give each worker its own isolated data. The wrong fix, the one the agent reaches for first, is loosening the engine back to the comfortable lie. When a stricter backend breaks where a lenient one passed, the strict one is usually the one telling the truth.

Production-honest invariants: fix the source, never the check

A bug in the demo data once flooded a flow-tracing check with thousands of false alarms: "orphan" records, chain steps that looked like they never completed. The cause was in the demo data, not the check. The synthetic balance-maintenance legs (scaffolding that keeps the starting balances positive so the demo isn't all overdrafts) had been tagged with real money-movement rail names, so each one looked like the start of a money chain that never finished. One rail alone produced 2,619 phantom orphans. The tempting fix is to filter the check: skip the records that look synthetic. That's a slow acting poison. It teaches operators the check has holes in it, and the next REAL orphan slips through the exact same hole.

The right fix was in the demo data: give the scaffolding its own label-only rail that isn't the start of a chain, so it stops pretending to be real money movement. 2,619 false orphans dropped to 4 real ones, and the production check was never touched. The test I use: would this filter still make sense if a real institution's data fed this check? If no, it's a demo artifact, so fix the source and leave the production check honest.

Correcting errors without hiding them

Every reconciliation system eventually has to answer one question: when a posted entry is wrong, how do you fix it? The easy answer is to edit the row in place. Now the books are right and every trace they were ever wrong is gone. An auditor hates that, and they should.

recon-gen's ledger is append-only, you never edit a bad entry. You SUPERSEDE it: you append a new, higher-versioned row for the same logical record that carries a typed reason for the change. The original stays in the table forever. Operators see the corrected number, and a dedicated audit view (and the regulator-ready PDF) shows the whole chain: every prior value, the reason for each rewrite and a flag on any rewrite that showed up without one. Fix it in the open and prove you did.

That feature has been in the design since the beginning. The model was never the hard part, the CHECK was, and this is the honest part of the story. The spec says "errors may be corrected but not hidden, every correction must record why." I cannot mechanically derive a SQL check from that English sentence. Turning an English rule into a provably sound query across three database dialects (rather than one I've merely tested and believe is right) is an open problem, I track that work in the repo's plan as Phase DS. So today the detection is hand-written, and for a while it was wrong: it flagged ordinary entries as unexplained corrections. What caught it wasn't me re-reading the code. It was me doing UAT, adding addition tests to the test layer, planted-correction fixtures and the agreement between the different views failing until the check matched reality.

Where it all compounds: the four-way agreement test

The platform compiles one typed spec into three runtimes across three SQL dialects. The release-gating test reads four INDEPENDENT producers of the same numbers (the database directly, a PDF derived from the database, the self-hosted app and the live browser dashboard) and asserts they all report the SAME invariant violations. For the flat invariants it demands the exact same set of records, for the aggregated PDF it demands the counts agree within bounds.

The audit PDF and the live dashboards MUST agree on every number they both show. If they disagree, trust in the entire generator collapses and the work is useless.

A drift the dashboard shows has to equal the drift the PDF prints. And the test wiring that says "these four producers must agree" is checked before the tests even run, so renaming a producer fails loudly right away instead of quietly detaching the check, which would leave a passing gate that proves nothing.

This is where every principle stacks on the last. Typed shapes, gates that test themselves, no deferrals, the strict path as the trusted one and then on top of all of it four independent witnesses that have to tell the same story before a release ships. At that point the trust isn't something I hope for, it's built in.

None of this is autopilot

I should be honest about something, because the story so far could read like I built a cage of lints and tests and then sat back while the AI filled it in. That's not what happens.

Here's the part that still surprises me: I've written MORE since I started working this way than in any stretch of my career. Specs, invariants, the rules themselves, the reasons behind them. Just like a junior dev, you can tell an agent to build something and it will but its 100% not what you meant.

While all this machinery is going up I'm in the loop the entire time, running the testing, making the calls on what the product should BE, arbitrating the architecture, deciding what's worth building and what's a dead end. It's the same product-and-architect work I've always done, just concentrated and moving faster than I ever have. recon-gen is on its 14th major version. It is, honestly, INTENSE, and it's also the most empowering way I've ever worked.

That's the real job of the guardrails and the part I'd be lying to leave out: they don't replace my judgment, they durably enforce it. Every lint, every typed invariant, every agreement test is one more thing I no longer have to hold in my head or re-check by hand. They make the ground behind me solid so I can keep sprinting forward without stopping to check it's still there. The trust is what buys the speed.

Without it, going this fast is just going off a cliff faster.

Why this is the skill now

The volume problem is solved. An agent will write you the code. The TRUST problem got harder, because the plausible-looking wrong answer now arrives faster and in far greater quantity than ever, and the thing producing it is genuinely good at making a failure turn into a pass.

So the long term skill isn't writing the next function, the agent does that now. It's building the system that proves the work correct, and then having the judgment to spend the speed that system buys you. Types that fail at the buggy line, gates that get tested on whether they still fire, a no-defer rule that drags the root cause into the same commit. That's the half a machine still can't own for you.

The faster the AI gets, the more leverage that control system has. You don't out-type the model, you build the rails it can't quietly step off and then you let it iterate!

That iteration is the real win, any human team would have revolted at the amount of change this code base has gone through but that iteration makes the final destination far better than if we just reached a local maximum.

recon-gen is open source (GitHub, on PyPI, with live demos running over a fictional "Sasquatch" institution). It's one deep proof point: a published product where I could practice this control system end to end and write it down as I went, not something deployed inside a real bank. I didn't beat the AI at writing code, I built the system that makes its code trustworthy.

Go poke at the repo, steal the lints that fit and tell me where I'm wrong, I'd genuinely rather hear it.

Cover image from PeterPan23, Public domain, via Wikimedia Commons