Death by Papercut using QuickSight
Companion to AI guardrails. This one's about a different kind of guardrail: the ones you have to build around someone else's product.
I build recon-gen, an open-source financial-validation platform. One of its release targets is Amazon QuickSight. I picked it deliberately, integrated it carefully, debugged it deeply and documented every defect I found with a minimal repro and a suggested fix. I'm about to rip it out.
I didn't start out this way, the initial build of recon-gen was solely focused on QuickSight: it was API first and had a nice WYSIWYG editor. The papercuts just grew so fast I couldn't ignore them and it resulted in the build of the htmx based renderer.
Papercut #1: Technical limitations
recon-gen generates the same dashboards into four runtimes: QuickSight, a self-hosted HTMX renderer, an audit PDF and direct SQL. A release doesn't ship unless all four AGREE (a 4-way agreement test gates every build). That architecture isn't an accident. The README tagline for the self-hosted renderer is, verbatim:
feature parity with QuickSight, minus the QuickSight bugs
That's not marketing copy. It's enforced by many tests. Part of why a second renderer exists at all is to have somewhere to land when QuickSight can't be made correct.
I kept a catalog: 23 published quirks (846 lines), plus a deeper engineering log, written as defect reports — each with a minimal reproduction and a suggested fix. I debugged this product more thoroughly than some of its own docs do. So when I call something a defect, I can hand you the reproduction.
The worst footgun. URL-parameter writes don't reach the dataset SQL layer. So a cross-app drill renders the UNFILTERED universe of data underneath a control that explicitly says it's filtered. The number on screen is wrong, and the UI is actively lying about why. I tried three documented workaround shapes and wasted multiple weekends with nothing to show. I dropped the cross-dashboard drill feature for Quicksight. The lesson I wrote down: assume the destination control will lie, and design the UX around that. You should never have to plan a UX around your BI tool silently showing the wrong rows — but if you target QuickSight, that's the planning you're signing up for.
Silently wrong numbers. QuickSight's CategoricalMeasureField COUNT silently renders as a DISTINCT count when the same column also appears as a dimension. No error. No warning. Just a different, wrong number. The fix was a literal-1 calculated field summed with SUM — a trick I shouldn't need, to make a count actually count.
Failures with no signal. A dashboard that hangs on a spinner forever, no error. A 32-element ceiling on ParameterDropDownControl that doesn't surface in dev and crashes the DEPLOY at customer scale — I had to write an entire picker_datasets.py module to escape it, then add a lint (no-l2-derived-static-values, commit 86e1fc24) so I can't accidentally reintroduce the pattern. KPI conditional-formatting landmines (uppercase hex required here, lowercase aggregation function required there) discovered only at deploy time. A DATETIME-bound-to-string zero-row bug (commit 20ce511d) where QuickSight coerces a parameter to full-precision timestamp text and silently returns nothing, with correct SQL on both sides.
I want to be fair here, because fairness is the point. QuickSight ships a working runtime. All four dashboards deploy and render on it. The things in my parity registry marked HARD_DIVERGENCE are structural — the Studio editor, the ETL probes — features QuickSight was never meant to host. Those are htmx only, not bugs, and I don't count them. I also don't count the minor paper cuts: the dropdown that only opens on a grey bar, the hidden single-character sheet tabs, boto3 typing-stub noise (that's pyright, not QuickSight) and "matviews don't auto-refresh" — which my own doc flags as Postgres/Oracle behavior, NOT a QuickSight defect. And a chunk of my e2e pain — WebSocket networkidle never firing, tables virtualizing ~10 DOM rows — is honest test-automation friction, not a correctness defect. The embed surface is just genuinely hard to drive deterministically.
Strip all that away and you still have a real list: silently-wrong counts, controls that lie about filtering, hangs with no error, deploy-time-only failures. Those are the ones that cost me my most valuable resource, TIME.
The governed part. Here's what I'm actually proud of. The pure quirk-workaround code is surgically isolated — about 700 lines, fenced off from everything else. Divergences between runtimes aren't ad-hoc; they live in a machine-checked parity-divergence registry (common/parity/breaks.py, 298 lines, lint-asserted, three severity classes). There's also a ~4,000-line browser/e2e harness that exists because QuickSight embeds have to be driven through a real browser to test at all. (To be clear: the ~29k-line HTML renderer is a full web app with its own Studio editor — most of it isn't anti-QuickSight scaffolding. The QuickSight-specific footprint is the ~700 quirk-lines plus that browser harness.)
To be clear QuickSight is not broken BUT "QuickSight is correct only if you build a second renderer to check it against, a registry to track where it diverges, a lint wall to stop yourself re-stepping on the mines and a browser harness to test the parts that can't be tested any other way." Good Luck!
Papercut #2: The Bill
Then there's the part that doesn't show up in any defect report.
AWS auto-provisions new users and accounts at the PRO tier. The moment you do, a silent 30-day clock starts toward a mandatory ~$250/month AI-infrastructure fee I never opted into. Not a trial I signed up for — a default I was enrolled in. Also unlike EVERY other AWS service, this fee is not metered, if you trigger it you are paying it until the end of the month!
The trap is the credits. New accounts get $200 in credits, and those credits hide the true cost in the billing dashboards until they're burned. Which is to say: they suppress the cost signal during exactly the window when you'd otherwise notice the meter running and bail. By the time the dashboard shows you real numbers, the 30-day clock has done its work.
And the GUI cost settings lie. When I went to confirm I'd actually turned the charges off, the console told me one thing and I didn't believe it. So I wrote custom API-verification code to PROVE the charges were gone rather than take the console's word. Then, to be certain QuickSight was the only thing that could be billing me, I ripped EVERY database out of AWS to isolate it as the sole possible cost source.
The damage, from Cost Explorer: $192.95 real out-of-pocket (as of this writing), on top of the $200 in credits I burned. Let me be scrupulous about that number, because fairness cuts both ways. Some of it I CHOSE: I was spinning up real RDS test databases, a legitimate cost I was happy to run on credits while I built. That part isn't a grievance — I'd have gladly kept paying it. The grievance is what the credits got actually drained by. The bill is 16 services, but the shape is a single concentrated May spike — QuickSight plus the Bedrock AI-infra the auto-PRO tier lit up — dropping to a single author user in June after the rip-out. The databases aren't what put me underwater. The fee I never asked for is.
To be precise: the ~$250/month is the mandatory fee the PRO clock leads to. The only reason why it wasn't the full $250 was due to the 30 day trial ending mid month. Here's the part that stings — my credits were comfortably covering the usage I'd deliberately chosen. The surprise auto-PRO enrollment is what drained them and tipped me into paying out of pocket. Without it, my out-of-pocket to date would have been $0.
It compounds: recon-gen's own CI deploys real QuickSight resources per run via OIDC, then sweeps them (everything's tagged), with a nightly nuke script as backstop. Running my own test suite literally costs AWS money. I built the kill switch and the cost-audit tooling because of this episode — the verification habit became infrastructure.
Papercut #3: Support? What Support?!
When I hit all the cost issues, I did the obvious thing and opened a support case. AWS advertises everyone gets support (and to my memory it USED to say 1 business day SLA).
Basic Support, the free default: it covers "billing and service quota related questions only." My cost dispute was exactly that.
As of today, 2026-06-25, I've waited 37 days. No response other than an AI reply two days after the case was opened telling me to do what I already tried and identified didn't work. Case #177914468400391.
There's nothing more to say about this one, which is the problem.
What this actually cost
I'm going to rip QuickSight out of recon-gen entirely soon™. Someone can always fork the old version if they want to keep targeting it. The four-runtime architecture means removing it is a deletion, not a rewrite — the dashboards already render correctly three other ways, and that was the whole point.
There's a structural point that's bigger than my $167 May Bill. That mandatory ~$250/month isn't a usage charge you can throttle down. It's a trap and has been made even harder to avoid with the move to Amazon Quick. That's on top of the mandatory user fees that no free or hobby tier can avoid. Which puts QuickSight categorically off-limits to open source: no unfunded project can carry a quarter-thousand-dollar monthly baseline just to keep a demo alive. I did everything right — kill switch, tagged sweeps, ripped the databases out — and the trap still cost me. So the self-hosted renderer I keep calling a "second runtime" isn't really a luxury. It's the only reason recon-gen can exist as open source at all long term.
The sticker price of a closed vendor tool is the smallest number in the deal. The costs that matter never show up on the pricing page — you only find them by building a second implementation to disagree with the first. That's the part that should change how you pick a tool.
For me, personally, the verdict is narrower and permanent: the trust is gone for anything low-stakes. I was fine paying for what I chose. I'm not fine being defaulted into a fee I didn't ask for, having it hidden behind credits, lied to about by the console, then ignored for 37 days when I asked for help. I won't build on AWS again for a personal project. If I ever do, it'll be with a business and a revenue stream big enough to make that surprise-fee exposure a rounding error — which is, I suspect, exactly the customer it's built for.
What let me write this with receipts instead of feelings is the same thing recon-gen is built on: generate it yourself, and verify it. If recon-gen had only ever targeted QuickSight, I'd have shipped wrong counts under controls that lied about filtering, and I'd have no idea. Because it targets four runtimes that have to agree, the bugs had nowhere to hide. Neither, in the end, did the bill.
If you're weighing QuickSight (or any closed BI tool) for something that can't absorb a surprise floor, I hope the line items here save you the days I spent finding them.
recon-gen is open source: github.com/chotchki/recon-gen. The quirks catalog and parity registry are in the repo if you want the full forensic version.
Cost Explorer: the May spike — RDS test databases plus QuickSight — drops to a near-flat QuickSight-only line through June, once I ripped it out.
Cover image from Yinan Chen, Public Domain, via Wikimedia Commons
