An API key gone rogue and ran up $10k on my Google Cloud in 3 days. It was the second time. Go audit your keys.

Let me start with the villain. Anonymous attackers abuse the Gemini API with floods of generation requests. I still do not know exactly how they do it, or why. Google detects this, defends against it, and in my case handled the whole mess professionally in the end. The problem is what this class of attack can do to a founder while the good guys work through it at their own speed.

Here is what happened, twice, and what I built so a third time can't sink me.

The honest admission first. I do not have a disciplined habit of going into my cloud console every month to review the API keys I've created. I never audited mine. So a key named "API Key 1" – a default name I would never type myself – sat there unnoticed. It had been created "unrestricted" by default through the Cloud Console, I never deployed it in any app, and it had zero usage for eight months before it was abused.

The leak was (as we concluded) on Google's side – that key string only ever lived in their systems. But I'm the one who never went in and deleted a key I didn't recognize. If you take one thing from this post, take that: open your console today and look.

This was the second incident, and the two together are the real story. In August last year, a similar attack charged a large sum – tens of thousands of ILS – to my card. Google investigated, agreed it was not my usage, and credited it back after a few weeks. Not as cash: as balance on my cloud account.

I chose to keep it there and spend it down on normal usage, which felt reasonable at the time. Then this May, the second attack ran up $10k+ in three days – 1.45 million calls, 65 million image generations from that one key. It ate my entire remaining balance from the first incident, and then created a fresh debt on top.

Then my whole account went dark. This part matters.
At that exact moment, my payment card was blocked because I was in the middle of switching which cards I use to pay for services. So when Google tried to charge the fraud-created debt to that dead card, it failed. And a failed charge auto-suspends the entire billing account: every project offline, including ones with nothing to do with the incident, plus a 30-day notice that it would all be deleted.

Now picture the corner I was in. Anonymous attackers had used my key to rack up several thousand dollars of debt with their own generation requests. That debt first swallowed the entire balance still sitting in my Google account from the August incident – wiped out in a few days. And Google now needed me to attach my new bank card and let it pull several thousand dollars more. Until I did, the whole account stayed frozen – every project, every service, all of it dark.

So there was no move left to make. I was a hostage with no exit. The only way to switch the lights back on was to attach that card, knowing that the moment I did, several thousand dollars would be pulled straight off it – money I had no guarantee I would ever see again, that I would then have to fight for, for weeks, with no promise it would come back.

Pay upfront for a crime I didn't commit, and then fight for your money to get it back. Which was exactly what I did in the end.

The part founders underrate: the speed, and how much time, explanations and effort it costs you. Google got there in the end, after weeks of escalations and 'support-ticket-soccer' – once the right team looked, they reversed every charge and returned me all the money in full, professionally. Back to my GCP account balance. Which I immediately decided to grab back to my bank.
In full. Nothing personal.

But the investigation and refund took about six weeks. Sit with that.
For six weeks, thousands of my money sat as an interest-free credit with my vendor while a dispute I was always going to win crawled through the queue.

Now scale the number. If a leaked key runs up $100,000 before anyone catches it, and it takes six weeks of back-and-forth to claw it back (which is NOT guaranteed), that is no longer an annoyance – that can put a person on the edge of personal bankruptcy. That is the real stake here: maybe don't let one vendor be a single point of failure on your solvency.

Two small tactics, since the process is slow and you'll be doing some of the work yourself.

1. Google's automatic "cost anomaly" email put the damage well below the real figure, so I pulled the full cost-table export and computed the true number myself – never anchor on the number they hand you.

2. When part of the GCP balance refund stalled on a "there is no payment associated with that amount" ruling, I had to stop arguing definitions and quoted Google's own earlier statement back to them: a section headed "PAYMENTS RECEIVED," plus arithmetic that reconciled to the last agora. The refund request was raised the next day. So keep every statement they send you.

Final thing: a cloud "budget" is not a spending cap. This is the one I got wrong too. A Google Cloud budget does nothing to stop spend – it emails you at 50/90/100% and that's it. A receipt, not a fuse.

Real protection is two layers you build yourself:

Real-time quota caps on the expensive API. Cap the project-wide request rate, not one model – this attack spread its allowance across dozens of models. A tight cap turns "unbounded" into "pocket change," and it triggers instantly.
A kill-switch. A budget publishes to a Pub/Sub topic; a small Cloud Function subscribed to it disables billing across all projects when spend crosses a hard ceiling. When it fires, everything goes offline.

That is the point: offline beats bankrupt. Deploy it in dry-run first so it logs what it would kill without touching anything.

The structural fact underneath: a leaked key on Google Cloud can run genuinely unbounded spend. Other AI vendors hand you a hard prepaid cap by default but looks like GCP does not. That difference is exactly why you build your own circuit breaker instead of trusting the platform to stop the bleeding.

So, today: audit every API key in your console and delete what you don't recognize; replace the word "budget" with "alert" and build a real cap.

Google is doing its best but at six-week resolution speed, the only person who can protect your runway in the meantime is you.