Building in the AI reliability space — looking for feedback
Hey everyone 👋
I'm exploring a problem around AI customer support: teams can deploy an AI chatbot successfully, but understanding its quality across real customer conversations can become difficult as usage grows.
I've been talking to AI founders and collecting different perspectives on how they currently evaluate their systems.
My next step is to build a very small prototype around one specific problem I've identified from these conversations.
I'm intentionally keeping the first version simple rather than trying to build a huge platform.
I'd love to hear from other Indie Hackers:
If you were building an AI customer-support product, what would you want an evaluation tool to tell you every day?
Would really appreciate your thoughts, especially from people who have built or deployed AI products.
honestly for me it'd be less about raw metrics and more about "would this response have made me trust the company less." things like, did it actually solve the person's problem or just sound confident while dodging it, did it know when to hand off to a human instead of guessing, and consistency over time (not just one good convo, but not drifting into bad answers after a few hundred conversations). the trust piece is the hard one to quantify but it's usually what actually gets escalated to a founder's inbox lol. curious what you're hearing most from the AI founders you've talked to so far, is it mostly accuracy issues or more the "doesn't know when to stop" problem?
What you're describing sounds less like an AI reliability problem and more like an evaluation problem that only becomes obvious once teams have enough real conversations. I'd be curious whether the prototype is centered on finding failures, measuring quality over time, or helping teams decide what to fix first.