Recovery detection missed one case and my incidents never closed

Building an uptime monitor, and the check engine keeps humbling me.

Recovery detection looked fine:

if (lastPing.status === 'up' && prevPing.status === 'down') {
  closeIncident()
}

Then a site went down, came back slow, then came back fine. So the sequence was down → degraded → up. Previous ping was degraded, not down, condition never fired, incident stayed open. Dashboard showed the monitor green with an open incident hanging off it.

Fix is prevPing.status !== 'up'. One character. Took me three days to spot because everything looked green.

The other one that needed a rewrite was re-alerting while something stays down. If you want a reminder every 30 min you enqueue a delayed job, and there are about four ways that turns into 40 emails. Landed on three guards, each covering a different hole:

Redis lock with TTL equal to the renotify interval, so a second job can't get scheduled for the same monitor
check for an open incident when the job actually fires, since recovery may have happened during the delay
delete the lock on recovery, so the next incident schedules clean

Any one alone leaves something open. The lock doesn't save you if recovery lands after it expires and the job's already queued. The incident check doesn't stop duplicate jobs piling up in the first place. Deleting a lock you never took does nothing.

Nothing clever going on here. Just that most of the work in "tell someone their site is down" turned out to be in the not-telling.

What I'm building is Dumza (dumza.com), flat-priced uptime monitoring. Free is 10 monitors at 5 min, $7 gets 1-minute checks. Email and webhook alerts only. Started it because every monitor I tried had turned into an incident management platform.

Happy to go into the worker setup if anyone's doing something similar. Bun + BullMQ on a small Hetzner box, Redis stays private to the worker, Postgres for anything the web app touches.