Messed up bad yesterday and caused an outage for my customers websites, but also got a surprise

Just want to share a quick story about the responsibility that comes with building apps.

One of my apps is a Blockade, basically an easy to use firewall for Shopify stores.

It processes tens of millions of visitors a month, and yesterday a 1 character typo cost my customers about 10 minutes worth of traffic across almost a thousand stores.

Once I realized the mistake, I immediately shut down the app's servers to stop the bug.

Then I wrote a quick email blast to explain the situtation, by now there were tickets flooding into my chat & email from angry merchants - totally understandable by the way.

About half an hour later I had a bugfix, tests & a deploy to staging where I verified the bug was fixed.

I launched the fix to production, and sent a follow up email explaining what happened, our response and what we're going to do to fix the issue going forward.

But what surprised me the most were the words of encouragement from some of my customers:

| It was really good of you to inform. i work in media affairs and it was otherwise well handled. good job 🌟

| You're doing a great job, thanks for the professionalism by letting us exactly know what happened

I lost about 5 paying customers from the outage and email blast so far (~$20 MMR) but it was worth being upfront with the whole customer base because it also earned trust with my other customers that stuck around.

Overall, I'm thinking that's my signal to start writing E2E tests instead of just unit & integration tests on my API only ha.