Developers January 18, 2020

What are some of your biggest bugs or mistakes?

Eric Turner @CodeOfTheProgrammer

Story time! I love hearing horror stories about when things go wrong. It's a morbid curiosity. ;)

Here are a couple of mine. Feel free to share yours!

My very first job in IT was for a small newspaper, and I was tasked with a mail merge to send out letters to all of our approximately 50k subscribers. We used an elaborate envelope stuffing machine, and sorted everything in a way to get the biggest postal discounts we could. So I had to query a database, output the results to a spreadsheet, run the mail merge, print the envelopes, print the letters (in the same order as the envelopes), and insert the printed letters and envelopes into the stuffing machine which would then tri-fold the letters, stuff the envelopes, and seal them. Well, I printed the envelopes and letters in a different order, so the letters were all delivered to the wrong people!

Another instance, I worked for a large enterprise. One of my jobs was writing and maintaining a browser extension for the employees to help them use information across multiple internal company web apps. We wanted to know the usage patterns of the extension, so whenever a user visited a page for one of the internal company web apps I would post an event to the corporate metrics server (maintained by another group). I messed up the code that determines if the user is on an internal company web app, so it started recording events for every web page that every user ever visited. The org that managed the metrics service had a hell of a time figuring out the sources of the DDoS attack!

  1. 4

    I somehow had an infinite loop in my code that sent 50,000 text messages to one person, in 10 minutes, and cost me like $350. The best part was the text was a Happy Birthday message that I can only imagine DDOSed someone's phone for a while :)

    1. 2

      LOL!

      Yea, when smart phones really took off somebody at work thought it would be a great idea to start sending SMS alerts for support tickets. Until something similar happened and I ended up with a huge bill (back before SMS was free. Remember the plans were you would only get X number of texts for free?)

      1. 2

        Oh yikes. So not only a bill from the dev side but everyone getting em eh? That’s brutal haha

  2. 2

    I can't remember what exactly caused it, but last year a glitch caused my API to repeat a $70 charge 13 times. It would have done more if Stripe didn't rate limit it.

    Fortunately, I noticed it, refunded all the duplicates, and emailed the user within minutes. Before they even noticed that they'd been charged $1K.

    Stripe doesn't refund fees though. So it cost me about $20.

  3. 2

    On my first job only been there a couple of months and a customer had a bug with a stats package. We couldn't replicate it on our Dev server to investigate so I had to back up the database and download a local copy. The database was huge so I had to take the backups in chunks (it was a myisam database so was having archive to zip several files take files in chunks instead of nysqldump as it was quicker).

    To stop the server running low on space after I downloaded the archive locally had to delete them. Problem was I did rm * (on a Linux server) thinking I was in the archive folder I created but I wasn't I was in the actual database directory so deleted the database files from under mysql. Ironically that was the last backup archive I needed so I was finished had hadn't realised my mistake.

    Customer isn't very on top of monitoring so took then a few weeks to notice but when they did every one was kinda confused s to what happened, we couldn't check our app alarms page as that had several million alarms and the pages couldn't cope missing them.

    Thankfully they didn't lose anything as i had the files I had backed up so was able to re upload and restore the database but it took a couple of weeks for the backlog of stats to get processed.

    1. 2

      Not quite as bad as missing the "." in a "rm -R ./*"
      Not a fun day!

      1. 2

        A colleague did do that while inside the /sbin folder, accidently did rm * -R thinking he was somewhere else. It totally binned the server as it couldn't log into it as it couldn't find ssh and other standard commands like ll and cd lol. Thankfully it was only one of our test servers not a customers so could have been worse, needed to re-install it thought.

  4. 2

    Oh man, those guys must have come after you with pitch forks! lol!

Recommended Posts