I stopped watching my agents. Here's what broke at 3am.

Hey IH 👋

Quick context: I'm a solo founder building Rapid Claw (https://rapidclaw.dev/pricing) with a small crew of agents — I usually run about 5 at a time, max. My brother Brandon handles most of the gnarly infra; I drive product and keep the agents pointed at useful work.

For the first few months I babysat every run. Staring at logs. Sanity-checking each PR. Killing anything that looked sideways. Comforting, but not sustainable if the whole pitch is "autonomous."

So a couple weeks ago I finally did the scary thing: kicked off a multi-step run before bed and went to sleep.

Here's the short list of what actually broke, in the order I found it the next morning.

1. An agent retried a failing webhook ~400 times in 20 minutes
   Not malicious, just enthusiastic. No exponential backoff on that code path. Cost was tiny, but it would've been ugly at scale. Added jittered backoff and a hard cap per job. Boring fix, should've been there day one.

2. One agent ate most of my daily model budget by 2am
   It got stuck in a "rethink → rewrite → rethink" loop on a task with fuzzy acceptance criteria. The task wasn't even important. Lesson: if the success condition is vague, the agent will spend unbounded money trying to guess it. Now every long-running task needs a concrete "done looks like X" check before it's allowed to run unattended.

3. A silent failure in a queue worker, for 4 hours
   No alert fired because the worker didn't crash — it just stopped pulling jobs. Classic. Brandon added a heartbeat + "no progress in N minutes" alert, which honestly we should've had before letting anything run overnight. If you're thinking about doing this yourself, please steal that idea.

4. The one that actually worked perfectly
   An agent quietly chewed through a backlog of small cleanup tasks I'd been ignoring for a month. Woke up to a clean queue and a tidy diff. That was the moment it clicked for me that the point of this isn't "more code faster" — it's reclaiming the small stuff so I can stay on the parts only I can do.

A few honest takeaways

Overnight runs are a forcing function. Every weakness in your setup shows up in the logs by morning. Better to find them on a quiet night than during a customer demo.

Guardrails > cleverness. Budget caps, retry caps, heartbeats, timeouts. None of it is exciting, all of it is load-bearing. I put together a rough breakdown of what we actually run in production over here: https://rapidclaw.dev/blog/unattended-agent-runs — not a polished guide, just what's working for us this month.

Small crew, small blast radius. I'm still a boutique shop. ~5 agents at any given time is plenty for one human to supervise and steer. The temptation to 10x the fleet is real, but every extra agent is another thing to babysit until your guardrails are truly solid.

Anyway — not a "we hit $X MRR" post. Just one founder learning the unsexy parts of running agents without a safety net. If you've been through this, I'd love to hear what broke first for you, and how you alert on it now. And if you're weighing options in this space, I threw together a rough comparison of how we think about it vs. the usual suspects: https://rapidclaw.dev/compare — feedback welcome, more brutal the better.

—Tijo