1
0 Comments

The dashboard tile that was quietly costing us 167,000 reads a day

A few weeks ago I opened Firebase and saw a number that made me uncomfortable.

Almost a million reads a day.

Firebase could tell me the total.

What it couldn't tell me was why.

Was it a user feature?

A background job?

A bug?

A dashboard page nobody even used?

The deeper I dug, the stranger it got.

One analytics tile was responsible for 167,000 reads a day.

Another was responsible for 161,000.

One path in our ingest pipeline was consuming over a million reads a day on its own.

The frustrating part wasn't fixing it.

The frustrating part was finding it.

Every tool we looked at could tell us a query was running.

None could tell us which feature was responsible, which user triggered it, or whether the reads were coming from a person or a machine.

So we built a small internal tool.

Instead of instrumenting individual queries, it sits at the datastore SDK boundary and counts everything automatically.

Every read is attributed to:

  • the feature that spent it

  • the user that triggered it

  • the environment it came from

A few fixes later, we watched one path drop by 396,000 reads per day.

That's when we realised this wasn't just our problem.

So we open-sourced it.

It's called Buckets.

I'm curious: how are other founders tracking database costs today?

When your bill spikes, how do you figure out which feature actually caused it?

posted toAvatar for product Crossdeck
Crossdeck