I had AI audit my codebase six times. It found a money leak every single time.
I run a personal finance app solo. It syncs bank data through a vendor
with a billing model that deserves its own horror story: once you hold
an access token for a bank connection, you pay monthly per account,
whether you call the API or not. Even if every call errors. The meter
only stops when you explicitly revoke the token.
Which means every code path that can orphan a token is a small leak of
real money. Forever.
Over the last two weeks I ran six full audits with AI agents. Not one
audit came back clean. Every single one found at least one leak, and
each was in a spot the previous fix didn't reach:
- Customer cancels, the webhook gets lost, and their bank connections
bill me monthly for an ex-customer. Indefinitely. - A user leaves a shared household while the token revoke happens to
fail. Then the last member joins a different household, the old
household gets deleted, and the cascade destroys the stored token.
Now nothing can ever stop that meter, and no report will ever show
it. That one genuinely scared me. - Stripe marks a subscription "unpaid" and that one webhook gets
dropped. My stored status stays frozen at the last paid value, and
every cleanup job keyed off the frozen value, so they all skipped it. - The worst one flipped direction: a customer whose card died could
fix it, subscribe again, get charged monthly, and receive nothing.
My code refused their new subscription forever because the old one
still looked "live" at Stripe.
That last audit was 44 agents. It also caught something I didn't think
to worry about: my own test suite was lying. The test for bug #4's
exact scenario mocked Stripe returning "canceled" for the old
subscription. Real Stripe parks dead subscriptions as "unpaid" and
never changes them. So the test was green, CI was green, and
production would have refused a paying customer until the end of time.
What I actually learned:
The leaks are never on the happy path. All six were in the failure
path of a transition. A revoke that fails, a webhook that vanishes, a
delete during an outage. Normal testing exercises what should happen.
Money leaks live in what happens when the thing that should happen
doesn't.
One AI agreeing with itself is worthless. The setup that works for
me: one set of agents hunts, then every finding goes to an independent
agent whose only job is to prove it wrong, with access to the actual
code. In earlier rounds about a third of findings died there. What
survives is worth acting on. What one agent asserts alone is a rumor.
Make the fix prove itself. For the fixes, the agent restored the
old broken code, confirmed that exactly the new tests fail and nothing
else does, then put the fix back. Cheap to do, and it caught two tests
that were passing for the wrong reason.
Assume leak #7 exists. I stopped believing any audit is the last
one. There's now a nightly job that re-derives the billing invariant
from scratch, kills anything violating it, and emails me when it fires.
The email means a seam I don't know about leaked, and cost me a day of
spend instead of a silent month.
Six for six is not a great feeling about the previous five fixes, if
I'm honest. But paying for six audits beats discovering leak #4 from a
customer's angry email.
Anyone else running adversarial verification on AI output, or is
everyone still trusting single-pass reviews? Genuinely curious what
setups people have landed on.
The repeated-audit result is striking. The fact that each pass uncovered a different failure mode makes the limits of single-pass AI review very concrete.
Agreed, and the part that surprised me is why the passes kept finding different things. It wasn't better prompting each round. Two mechanics did the work.
First, the target moves. Every fix creates newly reachable states, one repair made a race condition possible that the previous audit could not have found, because the code path didn't exist until the fix did. The auditors even flagged the coupling ("fixing bug 1 makes bug 3's race reachable"). So auditing after each change matters more than auditing harder.
Second, lenses beat effort. The same model that came back clean on a billing pass found five real bugs when pointed at color semantics, a category I only added because a human noticed one suspicious green negative number on a screenshot. The audits inherit your curiosity. They don't supply it.
The thing repetition never solved, knowing when to stop. Six for six strongly suggests a seventh would find something too. At some point you're paying for reassurance rather than protection, and I don't have a principled stopping rule yet. Closest I've got is structural, the last two finds were caught in production within a day by an invariant checker anyway, so the audits are now racing a safety net instead of racing disaster. Curious if you've seen a better answer.
That stopping-rule question is the part I find most interesting. I’d be curious how you end up thinking about it as the pattern develops.
I'll try to report back. My honest guess at where it lands, the stopping rule won't be a number of passes, it'll be economic. You audit until the expected cost of the next undiscovered bug falls below the cost of the pass. For me that line moved the day a nightly invariant checker started catching leaks in production within 24 hours, the audits stopped being the last line of defense and became a way of testing the net. Next audit runs after my next feature wave. If it ever comes back empty for the first time, that's data too, and I'll write up whichever way it goes.
That economic framing is interesting. I’d be interested to hear how the next feature wave changes the picture. What’s the best email to reach you on?
support@kaivorallc.com
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.
Just got a Zarek audit for my PH launch — brutal but necessary. AI audits catch what we emotionally ignore. How much did the money leak cost you before you found it?
Fortunately, almost nothing. The audits ran in the first two weeks after launch, when the user count was one (me). So the leaks were caught as code paths, not as invoices. Total real damage was a few dollars on my own test connections.
The stakes were still easy to model, because of the billing shape: the vendor charges monthly for every connected account for as long as an access token exists. Doesn't matter whether you call the API. Doesn't matter whether the user still exists. The problem was never the size of the charge, it was the shape: no expiry, no visibility, no user left attached to notice. The scariest leak destroyed the token during a household merge, which meant nothing could ever stop or even find that charge again. At a thousand users, a two percent leak rate would be a slow bleed I'd genuinely never see.
That's my actual takeaway: audit while the leak costs nothing, because the same bug found later isn't bigger, it's just older and buried under real customers. What did the Zarek one catch on yours?
Great breakdown — the "audit while it's cheap" mindset is spot on.
Zarek caught 8 things on mine. The brutal ones:
• Fake "+23% CTR lift" stat with zero source (would've been roasted on PH in minutes)
• "CTR Score" naming was misleading — renamed to "Thumbnail Quality Score"
• Empty FAQ, no screenshots, Free/Pro copy was muddy
The fake stat was the scariest. I put it there because it "sounded right" — classic founder bias. Zarek didn't care about my feelings, which is exactly why it was worth it.
Launching Aug 25 now with clean positioning. Better to get punched in private than on Product Hunt.
What was the most expensive leak you caught post-launch?
Congrats on catching the fake stat before PH did. That instinct to write down what "sounds right" is real, and it isn't just founders, my test suite did the same thing. The billing tests mocked the payment provider returning "canceled" for dead subscriptions, because that sounded right. Real Stripe parks them as "unpaid" forever. Green tests, broken production, nobody lying on purpose.
On your question, none of the leaks got expensive in dollars, they were all caught while my user count was one. But the one that would have been most expensive flips the direction you'd expect. It wasn't my money leaking at all. A customer whose card failed and who later came back and resubscribed would have been charged every month for a subscription my code refused to honor, indefinitely. Vendor-side leaks cost margin. Charging a customer for nothing costs trust, a refund, and the angry post that eventually finds its way back to a thread like this one. That's the bug that made the whole audit habit feel cheap and highly worth it.
Good luck on the 25th!
This is exactly the kind of insight you only get after shipping. "Vendor-side leaks cost margin" — I'm stealing that line.
The mock data trap is real. We optimize for the test environment that makes us feel good, not the production environment that actually pays. Your Stripe example is perfect — "canceled" vs "unpaid" is a one-word difference that costs trust at scale.
The scariest part: you don't know what you don't know until someone external audits it. Zarek caught things I was blind to because I built them. Same with your billing tests — you wrote the logic, so "canceled" sounded right.
Appreciate the good luck wishes. If you're launching anything soon too, would love to return the favor — warm list exchange is the only marketing that doesn't feel like marketing.
What's your current project? Saw the AI audit tool in your bio — is that what you're shipping next?
I am actively working on shipping a mortgage amortization calculator to my site today. I noticed in my ads people were searching for one and clicking on my mortgage calculator which wasn't fully fulfilling those users needs. As a result it unlocked a new opportunity I will try to capture.
Smart move — building what people are already searching for instead of guessing. Ads data is the closest thing to reading users' minds.
Mortgage calculator + amortization is a perfect combo. The first gets the click, the second keeps them on the site longer. That's SEO gold.
Let me know when it ships — would love to see the final version. And I'll send you the PH link on Aug 24.
Thank You, it has shipped and you can find it here https://financewithoutfluff.com/calculators/amortization-schedule
So far it seems "building what people are already searching for instead of guessing. Ads data is the closest thing to reading users' minds." was spot on. I shipped it an hour ago and it currently already has 3 visitors from 72 impressions proving there was demand for it that I was able to capture looking through search results on my other calculators.
3 visitors from 72 impressions in the first hour — that's a 4.2% CTR from search alone. Most landing pages dream of that.
Your instinct was right: the demand was already there, you just had to build the door. Smart to ship fast instead of over-engineering.
Let me know how the first week goes — curious if the conversion holds. PH link goes out Aug 24.