Building AI Agents Control Tower: the observability layer the AI stack diagrams forget
Every "Modern AI Stack" diagram I see has boxes for LLMs, frameworks, vector DBs, evaluation - and nothing for "is this thing still working right now, in production, at 2am." Evaluation tells you if an agent passed a test before launch. Nothing tells you if it's silently broken after launch.
AI Agents Control Tower watches live agent traffic and catches the failures that don't throw errors: a tool call that returns empty instead of raising an exception, a loop that keeps burning tokens without making progress, an agent that quietly stops doing its job while still returning HTTP 200. It's the layer between "deployed" and "actually working."
Solo founder, self-funded, live in production since June. Dogfooding it on my own agents daily, which is how most of the detection patterns actually got built - from watching my own stuff break first.
Say something nice to bob78…
Post Comment
-
1Your distinction between “passed evaluation” and “actually working in production” is interesting. I'm exploring the boundary between the two right now. Once you detect a production failure, how do you feed it back into the pre-production evaluation loop? In other words: does your production monitoring automatically create new regression cases, or is that still a manual process?5 days ago Reply
-
1Really interesting point on silent failures. One thing I’ve been thinking about is whether observability is enough if you can’t reconstruct the exact data/context state the agent saw at execution time. Logs can tell you what happened, but if the underlying data changed afterward, debugging the decision itself becomes much harder. Do you also bind each run to a specific input or data state?13 days ago Reply
-
1Great question, and honestly you've put your finger on a real gap. Right now we capture output_summary, tokens, cost, and duration per execution, but not the input/data state the agent saw at that moment. So if something changes upstream afterward, you can't fully replay why a specific decision happened. We've actually got a schema half-built for exactly this (step-level input/output capture), just never finished wiring it up. Good news: we're committing it to the roadmap now. It'll ship as an opt-in per-agent capture, same pattern as our other detectors, so you can turn it on for the agents where debugging-by-replay actually matters to you. Appreciate you pushing on this, it's exactly the kind of feedback that shapes what we build next.13 days ago Reply
-
-
1The silent-failure point is exactly where this gets interesting. Observability can show that the agent called a tool, returned 200, looped, stopped, or produced an unexpected result. But for consequential actions I think there is another boundary: What was authorised → what the agent attempted → what actually happened downstream → what can be independently proven afterward. A control saying “blocked” isn't necessarily proof that nothing executed. Equally, a successful tool call isn't always proof that the intended external state exists. That's the boundary we're working on with OpsWatch. Your Control Tower looks complementary rather than competitive to me: operational visibility inside the agent system, with an independent assurance layer establishing whether consequential external outcomes actually matched the authorised action. I'd be interested in comparing the two models against one bounded production workflow. There may be a useful separation of responsibilities here.15 days ago Reply
-
1Really well put, and I think you've drawn the boundary in exactly the right place. We watch the agent side (did it loop, did it stop, did the output pass a correctness check), but a successful tool call on our end genuinely isn't proof that the real downstream state matches what was authorised, that's a different, independent claim, and OpsWatch sitting on that side makes complete sense as its own layer rather than something we should try to absorb. Complementary is the right word for it. I'd genuinely be up for comparing the two models against a real bounded workflow, that sounds like a useful way to actually test where the responsibility line sits rather than debating it in the abstract. Feel free to reach out directly whenever works for you.13 days ago Reply
-
1Thanks Bob, this is exactly the distinction we have been exploring. I think the key point is that a successful agent execution or tool call can prove something about the agent workflow itself, but it does not necessarily prove that the downstream state matches the intended authority, context, or conditions at the moment of consequence. That becomes a separate assurance question. This is why we have been positioning OpsWatch as a complementary layer rather than trying to replace agent observability or correctness checks. The question is not only "did the agent execute?" but also "what can independently be demonstrated about what was authorised, what actually occurred, and whether the resulting state can be trusted?" I agree that a bounded workflow comparison is the right way to test this properly rather than debating it in the abstract. I would be very interested in comparing the two models and seeing where the responsibility boundary sits in practice. Feel free to reach out directly at [jason@mcgillintelligence.com.au](mailto:jason@mcgillintelligence.com.au) and we can continue the discussion there. Thanks again for the thoughtful response.13 days ago Reply
-
-