My scary encounter with a cron job running on my application

I have a simple application which posts to social media sites like facebook on a schedule basis.

I slowly started realizing that all the posts are being posted twice. I initially blamed my code and spent hours figuring out the root cause but of no use.

After further investigation, I found out that the cron job which posts them to social media sites is running on two servers. This extra server is a dummy server which i started to migrate the application. But i left it behind to focus on it in future when i have more time. But i did not realize that the cron job is running on this server.

I immediately removed the cron job entry from the dummy server. Then i realized, If this server is running cron jobs, Then it might also be running the cron job which generates invoices and processes payments.

I checked the other cron entry and it is running as well. I got panicked because i might be double charging the customers. I checked the database and there are indeed duplicated invoices and both of them are marked as paid.

I jumped into stripe and didn't find any duplicate charges. To my relief, after some further investigation, This dummy server is running stripe sdk in test mode. So no actual charge is made to customer's credit cards.

If the dummy server wasn't running in test mode, Then i would be in more trouble as i would not know about the issue until a customer complains about a double charge. I ended up building a cron job monitoring tool to avoid these kind of issues in the future.

After this incident, This is what i am doing and i suggest everyone to do if your application has cron jobs running.

• I Make sure to stop any test servers i create immediately after i finished testing.

• If application is load balanced, I will double check and confirm that cron job is running only on one server

• I use a cron job monitoring tool so i get alerted when a cron job is duplicated or failed.

• I make sure to always use stripe test mode for anything other than prod. Even when we are migrating our app to a new server, I will leave stripe in test mode until the last minute.