How we stopped babysitting CI failures — 3 lessons from routing every failure through one queue
Problem:
CI went red, nobody knew for hours. Then someone investigated, someone filed a ticket, and someone fixed it. The biggest cost wasn’t the failure — it was the queue that didn’t exist.
We built one ops queue that turns failures into work coding agents can pick up: failure → ticket → agent → PR.
Three lessons:
-
Don’t build infrastructure the cloud already gives you.
Cloud Build already publishes events. A filtered Pub/Sub push for FAILURE, TIMEOUT, and INTERNAL_ERROR sends only failed builds to /webhooks/cloud_build. No polling and no changes to build steps. -
Dedupe at the edge.
Pub/Sub and webhooks are at-least-once. We key on the build or delivery ID and use an atomic cache write with an expiry. First delivery wins; redeliveries no-op. That simple guard saved more time than fancy triage logic. -
File a real ticket, not another log line.
A useful ticket includes a title, reproduction steps, actual and expected results, priority, and tags. An agent can act on that. The same shape works for build failures, webhook errors, and public support tickets, so one router can delegate the work.
The payoff is a boring, reliable loop: a failure becomes a ticket, an agent investigates, and a human reviews the PR.
How are you turning production failures into work your team or agents can pick up without manual triage?
The strongest part is the failure-to-work pipeline. Turning CI failures into structured tickets that agents can actually act on is more valuable than simply adding another monitoring layer.