I tried to break my own payment system this week. It broke three times.

Someone left a comment on a post I made about letting an AI agent buy its own GPU compute. His point stuck with me. Proving a payment settled isn't the same as proving what actually happened. I could show the money moved. I couldn't show why a request failed, or what my system actually said to the agent in that moment.

That bugged me enough to spend two days doing something I'd been putting off. Actually trying to break my own system on purpose instead of just trusting it worked because nothing had blown up yet.

It broke three times.

First, the thing he actually pointed at. My system only remembered the winning provider on a job, not the failed attempts before it. I built a proper audit trail. Every attempt now gets logged, verbatim, timestamped, and the log can't be edited or deleted after the fact. I tested that last part myself, tried to quietly edit a row, and the database refused. Good.

Then I found something I wasn't looking for. A request that got rejected for hitting a rate limit had still somehow charged the customer. Real money moved for a job that never ran. That's the kind of bug that sounds small until you imagine it happening to a real paying customer who now has to trust you to notice and make it right without them ever finding out on their own.

Fixed it. Then, because I was already in there, I stress tested the fix by firing a bunch of requests at the exact same time instead of one at a time. Found a second, sneakier version of the same bug that only shows up under real concurrent load. Fixed that too.

While I was at it I found a third thing. My pricing was a flat number that didn't move with real market prices. When live costs ticked up even slightly, on every provider at once, jobs would fail, but the charge still went through anyway. So I ripped out the fixed rate entirely and switched to real time pricing based on what things actually cost the moment someone asks.

None of this is glamorous. It's the unsexy part of building something people are supposed to trust with real money. But I'd rather find this stuff myself, on a random Tuesday with nobody watching, than have a customer find it for me.

Wrote the whole thing up, bugs and all. kilawattcloud.dev