I Was in the Data Center When SQL Slammer Hit

A firsthand account of the night the internet nearly broke — from a NOC tech in a basement in downtown Los Angeles.

There's a version of this story that gets told in Wikipedia articles and security post-mortems. Timestamps, packet counts, propagation rates. Clean. Clinical. Impressive in its own way.

This isn't that version.

This is the version from the basement.

A Normal Tuesday Night

I was working third shift at a web hosting company in downtown Los Angeles. The year was 2003. Our company hosted things the internet largely hated — massive amounts of spam, adult content, the full unglamorous spectrum of early-2000s web commerce. We also sold rack space to outside customers, some of whom were running perfectly legitimate operations. It was a different internet. Nobody was pretending otherwise.

Third shift had a rhythm to it. Watch movies. Keep an eye on the spam boxes to make sure mail was flowing. Monitor the graphs. It was the kind of work that was 90% tedium and 10% genuine crisis, and you learned to tell the difference pretty fast.

That Tuesday night started as tedium.

Then the graphs moved.

"Something Really Bad Is Going On"

It started quietly, the way these things always do. Servers in the shared data room began alerting. Backbone graphs started spiking. At first it read like a local problem — maybe a compromised box, maybe a misconfiguration somewhere. We called our network admin at home to take a look.
He couldn't get in remotely. The pipes were already too saturated to get a clean connection. He got in his car.
In the twenty minutes it took him to arrive, things got considerably worse. Customer calls started coming in. Alerts were stacking up faster than we could acknowledge them. Our outbound internet — the connections that let us reach the rest of the world — was effectively gone. We were an island.
After about an hour of working through the networking hardware, our admin pinpointed the traffic pattern. He knew what was causing it. The why was still a mystery.
I went out to the data center floor to get a status update. He looked up from the rack with an expression I still remember clearly.
"Something really bad is going on and I have no idea what it is."
When your network admin says that, you don't ask follow-up questions. You go back to the phones.

The Smoke Break Revelation

Somewhere around customer call number twenty, one of the techs from another DC in the building showed up at our door. The building housed several data centers — we'd share smoke breaks in the way people do when they work nights in basements and need something to do at 2am.
He poked his head in and invited us to a smoke break like usual at that time of night.
I told him I didn't think I could step away from what was happening.
He looked at me and said: "Man, the whole internet is down. Are you fixing it?"
No, I was not fixing it. Our network admin, to his credit, agreed — go smoke, you're not helping in here.
Outside, the techs from the other DCs were already comparing notes. Something SQL-related. Widespread. Every shop in the building was seeing the same thing. That was the moment the scope of it started to shift in your mind — from our problem to something else entirely.
It was January 2003. Sixteen months after September 11th. Nobody said it out loud, but nobody needed to. When customers called in and casually mentioned that their home internet was also running like dog shit, a few of them voiced the thought that was already floating around the room. More than a cyberattack. Maybe something coordinated. Maybe something worse.
We didn't know. We couldn't know. The internet was too broken to tell us.

The Grid Goes Red

After a while we got enough bandwidth back to load a single page. In those days there was a site that displayed a grid — five by five, maybe larger — showing ping times between all the major backbone providers. Global Crossing, Level 3, Qwest, all of them. Green was fine. Red was not fine.
We loaded the page.
It was all red.
Every backbone. Every direction. Every connection between every major provider on the map, red.
Nobody said anything for a moment. You didn't need to. The grid told you everything.

Slashdot and the Collective Fix

Eventually we blocked enough outbound traffic to get a reliable connection out. The first place we went was Slashdot — because in 2003, if something was happening to the internet, Slashdot knew about it.
The top pinned thread was about the outage.
It wasn't fully identified as the Slammer worm yet, but the SQL connection was already there. And in that thread were NOC techs and network admins from around the world — from their own basements and server rooms and data centers — posting in real time about what they were seeing and what was working.
I was fairly new to Slashdot at the time. I posted that we had fully blocked outbound SQL ports on our GSRs — the big Cisco 12000 chassis where our backbone connections terminated, the ones with line cards the size of cutting boards — and that after bouncing the boxes, our pipes had started to calm down.
I wasn't the only one suggesting it. But I was one of them. I remember getting upvotes. I remember the thread moving fast. I remember thinking that this was something I'd tell people about someday.
Our network admin put the fix in place. Within a few hours, the graphs started to settle. The phones quieted down. The grid, eventually, started turning green again.

What I Still Think About

The SQL Slammer worm infected roughly 75,000 servers in its first ten minutes of propagation. It doubled in size every 8.5 seconds. It was, at the time, the fastest spreading piece of malware ever recorded. The fix — blocking a single port at the network edge — was almost embarrassingly simple once you knew what you were looking for.

But what I think about most isn't the worm.

It's that Slashdot thread.

The internet was on fire. Barely functional. And the people responsible for keeping it running found each other in a public forum and started sharing what they knew, as fast as the half-broken network would allow. No corporate communications team. No official incident response framework. Just people who understood the problem, pooling information in real time, building the fix collectively.

You don't see that much anymore. Partly because events like this are rarer — infrastructure is more resilient now, in most ways. But partly because the internet itself is different. When Cloudflare has a bad day, half the web goes down simultaneously, and the fix comes from one company's internal engineering team. The problem is centralized. So is the solution.

There's something to be said for resilience through distribution. For infrastructure that doesn't have a single point of failure. For the value of having people who actually understand the systems they're running.

That's a different article, maybe.

But it's why I still think about that basement, and that grid full of red, and a Slashdot thread moving faster than the internet that was hosting it.

The author has been working in web infrastructure since before SQL Slammer. He runs AnchorHost, a managed hosting platform for SaaS founders who'd rather not find out what their infrastructure does at 3am.