I built a tiny AI widget that lets websites answer questions from their own content
I've been building Askably, an AI Q&A widget for websites.
The original idea was pretty simple:
A visitor lands on a website.
They have a question.
Instead of searching through 20 pages of documentation, they can just ask.
Askably indexes the website's knowledge and puts a small AI assistant directly on the site.
Something like:
Visitor
↓
"Do you offer refunds?"
↓
Askably
↓
Answer + source
But while building it, I realized that "put ChatGPT on your website" isn't really the interesting part.
The difficult part is everything around it:
Getting the right information
Keeping answers grounded
Showing citations
Knowing when the answer isn't in the knowledge base
Preventing the AI from confidently making things up
Keeping inference costs low
Turning conversations into actual leads
Giving businesses something useful instead of another generic chatbot
So I've been building Askably around those problems.
One thing I'm particularly interested in is knowing when NOT to answer.
If the website doesn't contain the answer, I'd rather have the assistant say that than hallucinate something just to keep the conversation going.
I'm still figuring out the product and distribution, so I'd love feedback from people who have actually used website chatbots.
What makes you uninstall/remove an AI chatbot from a website?
Is it:
Bad answers?
Too much UI?
Slow responses?
Hallucinations?
Generic answers?
Cost?
The chatbot just isn't useful?
I've put the current version here if anyone wants to play with it:
Would especially love feedback from other indie hackers building AI products.
YalanDeng is right that the "I don't know" list is part of the product. One thing I haven't seen in the thread: the widget reads site content, and the model can't reliably tell content from instructions. If a site has reviews or a comments section, a visitor can write text that the bot later follows as an instruction.
We ran into the write side of this with an AI agent in our own apps. Some text fields end up as instructions for another AI feature, so we flagged them, and any agent edit to them now needs a person to confirm. The read side is still open for us. Nothing yet stops other text from ending up in a prompt.
Do you index user-generated pages, and if so, do you treat them differently from pages the owner wrote?
That's a really good point, and honestly something I hadn't considered deeply enough. Right now Askably primarily indexes the site's accessible content, so I wouldn't want to assume that everything it retrieves is owner-authored or trustworthy. Treating user-generated content differently from owner-controlled content is probably something I need to add, especially for reviews/comments. I think separating content by trust level and never allowing retrieved text to act as instructions is the right direction. Definitely adding this to the security checklist.
Askably live while you’re still figuring product and distribution — and asking what makes people uninstall site chatbots — is such an honest early stall. Soft free text-only week for that first-site-that-keeps-it-installed freeze. Reply with what “kept installed a week” would look like in one sentence.
For me, “kept installed a week” means the site owner sees visitors actually using it to get answers they’d otherwise have searched for or asked support about.
The "ask instead of search" pattern solves a real problem. The failure mode I have seen most often with this kind of widget is when the content quality is inconsistent - the AI answers confidently from a mix of good and outdated docs, and the user ends up more confused than if they had just searched.
The quality bar question is hard to answer before you deploy but it matters a lot for trust. How are you handling confidence thresholds - do you surface a "we might not have a great answer" message, or does it always respond?
Yeah, this is exactly the failure mode I'm trying to avoid. Askably has an explicit "I don't know" path rather than forcing an answer when the retrieved content isn't strong enough. I'm also treating unanswered questions as useful product data because they usually point to gaps in the site's knowledge base.
An explicit abstain state could be measured as a conversion feature rather than only a safety feature. I would log the visitor question, retrieval confidence, citation click, and next action, then review unanswered questions weekly to improve the source content. That ties answer quality to business value without rewarding confident guesses.
I really like this framing. That's pretty much how I'm thinking about it too: abstention shouldn't just be a safety mechanism, it should generate useful data about what visitors actually want to know.
Grounded answers plus a clear abstain path seem like the right foundation. I’d track retrieval hit quality and abstention rate separately from answer latency, then add a small eval set of real support questions to catch regressions. Source snippets with document/version metadata would also make citation failures much easier to debug.
This is a really good point about separating retrieval quality from latency. I'm especially interested in building the eval set from real visitor questions rather than synthetic examples. The version metadata for citations is something I hadn't considered deeply enough.
A useful retention test is whether the answer helps the visitor complete a next step, not just whether it sounds fluent. I’d track cited-answer rate, honest “I don’t know” rate, and downstream actions separately. UX-wise, an unobtrusive launcher and a clear no-answer state would beat an unasked popup on mobile.
100% agree on the UX side. I'm trying to keep Askably as an unobtrusive launcher rather than something that jumps in front of the visitor. The goal is to help when someone actually has a question, not interrupt them.
A useful retention test is whether the answer helps the visitor complete a next step, not just whether it sounds fluent. I’d track cited-answer rate, honest “I don’t know” rate, and downstream actions separately. UX-wise, an unobtrusive launcher and a clear no-answer state would beat an unasked popup on mobile.
The “know when not to answer” rule is the product, not a footnote—especially when citations and cost are competing for the same token budget. I’d turn unanswered questions into a tiny content backlog; the chatbot can quietly become the least annoying product manager on the site 😄
😂 That's actually how I'm starting to think about it. The unanswered questions are basically a continuously generated content backlog for the business.
Have early users shown that answer reliability is what makes them keep Askably installed, or does the real retention trigger turn out to be the leads and actions generated from those conversations?
That's something I'm still trying to get enough data on. My hypothesis is that reliability gets people to keep it installed, while leads/actions are what ultimately make it valuable to the business. I'm trying not to assume that's true until I have more usage data.
That split between retention and business value is worth following. If you’re open to it, what’s the best email to reach you on?
Knowing when not to answer is the best feature on your list, and the unanswered questions are worth more than the answered ones. Every "that isn't on this site" is a page the business should write. A weekly export of those would sell to owners better than lead capture.
It connects to what we see with AEO at UtilitySEO. The questions visitors type into your widget are the same ones people ask ChatGPT and Perplexity. If the site can't answer them, AI search can't cite it either, so your gap log doubles as an AI visibility fix list.
On your question: as a visitor, I close chatbots that pop open unasked or cover the page on mobile. Bad answers come second.
One edge case: we run in 7 languages. Does Askably answer a Spanish question from English content, or say it doesn't know?
This is a really interesting way of looking at the unanswered questions. I hadn't thought about the gap between what visitors ask and what AI search might be looking for in quite that way, but it makes a lot of sense.
And yes, I'm with you on the UX. Personally I hate chatbots that automatically open or cover half the screen 😂
For the language question, Askably can handle questions in languages other than the source content, but I'm still working on how strict the grounding should be there. I'd rather give a properly grounded "I don't know" than confidently translate/reconstruct an answer that isn't actually supported by the source.