recon-gen is on its 14th major version. Not its 14th release — its 14th time I made a big enough change that semantic versioning required a major version bump due to its size. I did that in two months. Any human team I've been on would have revolted at that much rework, and they'd have been right to: at human cycle time, that's a death march (full disclosure it was kinda a death march on me).

That churn is the whole point, and it's the thing people don't understand about working with AI.

The pitch for AI coding is volume — it types faster than you. That's not the win it sounds like, there's no human way to read AND reason that fast. The part that actually changed how I work is cycle time: how long the loop takes from "what if" to a built, verified answer I trust. There's a book, Accelerate, that put hard numbers on it years before the current wave — the thing that best predicts whether a team ships well isn't talent or tooling, it's how short that loop is. AI didn't just make my code faster to write. It made the loop short enough that my desire to try was possible.

idea -> build: agent builds it (minutes to an hour)
build -> verify: ./run_tests.sh (seconds -> 45mins)
verify -> judge: I make the call
judge -> idea: throw it out, try the better one

When anything is expensive, "good enough" wins by default — not because it's right, but because the next attempt costs too much to justify. That is rational because time and money aren't infinite. So if I can try more with less time and cost, that's EXACTLY what I'm going to do (and have done)!

A short loop you can't trust is worthless

So the loop has to be fast AND honest. Making AI-written tests actually trustworthy is its own fight, and its own post (how I make AI write software I can trust). However AI didn't change the underlying test problem, it's still there, they have to be dead simple to run and quick enough that no one gets too impatient and starts skipping them.

Simple: one entry point. recon-gen has exactly one way to run its tests — ./run_tests.sh. Not a wiki page of commands, not "first export these five variables." One script and the defaults run it ALL (you can always narrow the tests easily). That matters more than it sounds: if running the check is a hassle, you (or the agent, under pressure) skip it — and a skipped check is a broken loop.

Fast: stop at the layer you're on. The suite is layered — units, then integration, then the full browser run — and ./run_tests.sh up_to=<layer> runs only up to the stage I care about. Iterating on a unit? I don't pay for the browser suite every lap. The whole truth is one command away when I need it; up_to gives me the relevant truth in seconds, which is what keeps a lap cheap enough to take another one.

direction: down

title: "recon-gen test pyramid — full chain ≈ 43 min" {
  shape: text
  style.font-size: 22
  style.bold: true
  near: top-center
}

stack: "up_to=<layer> — stop the run at any layer" {
  grid-rows: 6
  grid-gap: 12
  style.fill: "#ffffff"
  style.stroke: "#b8c2cc"
  style.font-size: 16
  style.bold: true
  style.font-color: "#c0392b"

  # qs_browser doubles as the legend: width = test count, height = time
  r1: "" { grid-columns: 2; grid-gap: 14; style.fill: "#f6f8fa"; style.stroke: "#d9e0e6"; style.border-radius: 6
    l: "qs_browser\n247 tests · 18.6m" { width: 150; shape: text; style.font-size: 13 }
    ba: "" { grid-rows: 2; grid-gap: 4; style.fill: transparent; style.stroke: transparent
      brow: "" { grid-columns: 2; grid-gap: 8; style.fill: transparent; style.stroke: transparent
        b: "" { width: 102; height: 153; style.fill: "#bcd9f0"; style.stroke: "#2c5f8a" }
        ty: "↕\ntime" { shape: text; style.font-size: 11; style.font-color: "#6b7785" }
      }
      tx: "↔ test count" { shape: text; style.font-size: 11; style.font-color: "#6b7785" }
    }
  }
  r2: "" { grid-columns: 2; grid-gap: 14; style.fill: "#f6f8fa"; style.stroke: "#d9e0e6"; style.border-radius: 6
    l: "qs_api\n45 tests · 99s" { width: 150; shape: text; style.font-size: 13 }
    b: "" { width: 43; height: 45; style.fill: "#cfe3f5"; style.stroke: "#2c5f8a" }
  }
  r3: "" { grid-columns: 2; grid-gap: 14; style.fill: "#f6f8fa"; style.stroke: "#d9e0e6"; style.border-radius: 6
    l: "deploy\ngate · 0 tests" { width: 150; shape: text; style.font-size: 12; style.italic: true }
    note: "(time not measured)" { shape: text; style.font-size: 11; style.italic: true; style.font-color: "#9aa7b4" }
  }
  r4: "" { grid-columns: 2; grid-gap: 14; style.fill: "#f6f8fa"; style.stroke: "#d9e0e6"; style.border-radius: 6
    l: "app2\n93 tests · 20.4m" { width: 150; shape: text; style.font-size: 13 }
    b: "" { width: 62; height: 160; style.fill: "#bcd9f0"; style.stroke: "#2c5f8a" }
  }
  r5: "" { grid-columns: 2; grid-gap: 14; style.fill: "#f6f8fa"; style.stroke: "#d9e0e6"; style.border-radius: 6
    l: "db\n109 tests · 44s" { width: 150; shape: text; style.font-size: 13 }
    b: "" { width: 67; height: 30; style.fill: "#cfe3f5"; style.stroke: "#2c5f8a" }
  }
  r6: "" { grid-columns: 3; grid-gap: 14; style.fill: "#f6f8fa"; style.stroke: "#d9e0e6"; style.border-radius: 6
    l: "unit\n5507 tests · 75s" { width: 150; shape: text; style.font-size: 13; style.bold: true }
    b: "" { width: 480; height: 40; style.fill: "#a9cdec"; style.stroke: "#2c5f8a" }
    p: "pyright · 0 errors\n682 files · 14s" { width: 169; height: 34; style.fill: "#e7f0d4"; style.stroke: "#5f7a2a"; style.font-size: 11 }
  }
}

Local and CI parity matter

Here's the gap most setups leave open. recon-gen's CI isn't a different world — it's the SAME ./run_tests.sh running on a beefy box I own (a dedicated self-hosted runner), so "passes locally" and "passes in CI" are the same.

The thing that secretly murders cycle time is the failure you only discover twenty minutes later, in some cloud runner that's running something DIFFERENT. Close that gap with identical scripts and you can push with confidence. Note: I do run using local self hosted runners because they are large and powerful and I don't have a company's size budget for runners.

The CI becomes gatekeeper to release, it's hard to bypass its checks AND you can't hide you did so. So ultimately you are incentivized to test early and often because if you don't, you still can't release!

This isn't new

Nothing I've said here is new, this is the same lean/agile/TDD/XP advice you would give any team no matter the AI usage or not. However what AI does is makes NOT having this painful. You will get immediately annoyed when something instantly breaks in a different place, when builds climb to over an hour so you can't iterate and worse you're more likely to hit your context window which reduces usable output.

That annoyance is a great sign for what to focus on to fix your problems and all the old tools work just fine to benchmark the pain. Even better now, you can iterate quickly to get back out of the hole you've dug for yourself.

Cover image from N. ellladee, CC0, via Wikimedia Commons