Why Coinbase crashed after their Super Bowl advertisement

In the recent Super Bowl, Coinbase launched one of the most genius ads of all time - resulting in 20 million site visits within the first minute of the ad airing.

However, there was one problem in this otherwise perfect marketing stunt: The site crashed - probably preventing an even bigger success.

While the whole thing was still a huge and major positive result for Coinbase, let's analyse the website crash and try to find some hints on how we would prevent such occurrences.

The Ad

Let's first look into the ad itself. The advertisement was a 14 Million Dollar QR code - no joke, the whole ad was only a QR code leading to their landing page.

Coinbase simply had a QR code displayed on-screen, nothing else. The plan was for everybody to scan the QR code, just to see what it was. So naturally, this ad will provide many many times more website visits compared to "normal" Super Bowl ads.

The High-Level Analyses

The site didn't crash due to an influx of new users - it was simply due to an increase in website visits. Why? Because most people scanned the QR code were redirected to the webpage/landing page.

I would go as far as to say that the database and their backend infrastructure were not involved at all. All speculation, but I guess at least somewhat reasonable. By the way, I'd love to see an incident report from Coinbase, this would be a great learning opportunity.

Prevention strategy

Ok, all of that being said and assuming Coinbase uses Ruby on Rails and React as they suggested, I'd create a scaling strategy as follows:

• Make sure the frontend is appropriately horizontal scalable. Either use one of the cloud providers auto-scaling options (Azure ACI or AWS ECS) or if you have virtual-machine-based deployments, you will have Kubernetes anyhow. Make sure to have autoscaling enabled for your frontend at least.
• Have a proper load balancing in place. I don't know the load balancing options for ACI or ECS by heart because we had our self-hosted HAProxy load-balancer back than. However I guess they all have a similar feature set. With HAProxy you have multiple settings to prevent server downtime. Having settings like maximum connections and throttling actually should guarantee that you don't run into downtimes. Dynamically change max connections with the number of horizontally scaled services.
• In the case of coinbase, I'd make sure to have at least some operations folks do manual service monitoring. Beforehand, prepare easy to install worst-case-mechanisms, like setting throttling or again max. connections, if the servers are hitting a limit.
• Make sure you have enough bandwidth available. This one is actually a big one. Depending on which cloud service you use, you have limited network (and disk!!) bandwidth. Do some calculations and make sure you have enough provisioned - or at least have the option to easily provision more bandwidth.
• Overprovision. I mean, look, it was clear, that a lot of users will hit coinbase during the superbowl ad. Make sure to have 100x or more of the resources you would normally have. If you are coinbase, your application (if it's at least somewhat state of the art) will easily scale to 100x and scale back 2 days after the superbowl without a massive increase in costs.
• I'd also look into from where my content is delivered. As the superbowl is a global event, I would definitely look into content delivery networks which can cache and serve my site around the globe. As you already hinted, cloudflare is one of the good solutions here.

Writing all of that, I realize it really comes down to planning. I'm sure coinbase has IT operations people who are smarter on more experienced than me. They of course would have known the above points. But I guess they were simply not expecting this HUGE load of traffic. Maybe they even were not informed about the nature of the advertisement they did. Coinbase simply had a QR code displayed on-screen, nothing else. This is genious from a marketing perspective, because everybody will scan the QR code, just to look what's behind. So naturally, this ad will provide many many times more website visits compared to "normal" superbowl ads. Maybe the IT operations simply calculated a normal superbowl ad conversion?

What do you think? Does my analyses as well as prevention point seem reasonable? Do you have any further tips for all of us to prevent website downtimes?

Resources:

• 20 Mio site visits in first minute: https://www.sportingnews.com/ca/nfl/news/coinbase-qr-code-commercial-super-bowl/i9yom8xdlzv1jcxs8oubusm7
• 14 Mio dollar costs for the ad: https://www.sportingnews.com/ca/nfl/news/coinbase-qr-code-commercial-super-bowl/i9yom8xdlzv1jcxs8oubusm7
• Coinbase using Ruby: https://blog.coinbase.com/tagged/ruby-on-rails