I Have an AI Agent That Tests My Own Product Every 3 Hours

I kept telling myself to “dogfood my own product more,” but the truth was simple: I wasn’t doing it consistently.

Right after shipping, I’d use the app a lot. Then I’d go back to building, and weeks later I’d realize I hadn’t really experienced the product as a user in a while.

So I set up an AI agent to do it for me.

Every 3 hours, it:

• checks my product via API
• opens the live UI in a real browser
• looks for broken flows, mobile issues, and weird empty states
• and if it finds something real, it can open a PR with a fix

What made this especially useful is that I later added a second loop with Sentry:

• dogfooding catches UX issues and broken flows
• Sentry catches production-only errors and hard-to-reproduce crashes

That combination turned it from “automated testing” into more of an automated maintenance loop.

A few things it caught that I probably wouldn’t have:

• API/UI data sync bugs
• mobile layout regressions
• blank empty states for new users
• production errors that only showed up in real usage
  One lesson: the AI is useful, but not magical.
  It’s good at repeatable checks. It’s bad at subjective judgment.
  And if you don’t add guardrails (tests, build checks, PR rules), it will confidently tell you it fixed things when it didn’t.

I wrote up the full setup here:
https://dev.to/toshipon/i-have-an-ai-agent-that-tests-my-own-product-every-3-hours-916

Curious if anyone else here has built something similar — not just AI for coding, but AI for continuously checking and improving a live product.