I built an AI visibility tracker, then stopped using it myself
I've been building on the internet for almost 20 years. Product, engineering, AI: I've done each of them hands-on, one step at a time. I've shipped SaaS for a traditional industry, for the pet industry, for generating game assets, and for game marketing materials.
The full-stack edge I was proud of (product design, online marketing, engineering) is slowly being diluted by AI. Anyone can ship now. How a product gets sold matters more than it ever did, and selling is my weak spot.
It showed worst with the traditional-industry product. I could barely get in front of the people who'd use it, and some days I honestly couldn't tell whether I was building the right thing.
So I started holding every idea to three rules:
The core of it has to be something I already know deeply and am good at.
I have to be one of its customers. With the traditional-industry product and the asset generators, I never was.
What it delivers needs a ruler. Good or bad can't depend on who's looking. With a ruler, I can point everything I know at moving the product toward "good".
Before Passcite I shipped a first version: an AEO tool that reported whether AI assistants mention your site, and kept tracking it.
Rule 2 is what caught it. I was its customer, and I didn't want to open it. It told me my AI visibility was bad. I knew that. What I needed was how to make it better, and it had nothing to say. A number you can't move is a number you stop looking at.
So I rebuilt the whole flow as Passcite:
Analysis: the buyer questions you lose across ChatGPT, Perplexity and Gemini, and who shows up instead on each one.
Why: what those answers cite, and what your own site never says.
How: every change, split in two. What we can draft for you, and what only you can do.
Action: you approve or reject each draft, tick off the rest, then re-run the same questions.
The split mattered more than I expected. A lot of what the assistants read isn't on your website at all. For every dental practice I've run it on, the answers cite Zocdoc and Healthgrades. A tool whose only output is a blog post has nowhere to put "claim your Healthgrades profile", so the biggest item falls off the list.
Rule 3 applies to me too, so here's my own reading. On September 21 I ran passcite.com through 30 buyer questions. 21 of them don't contain the word "Passcite", and on those 21, none of the three assistants mentioned Passcite. Not once. Every run since mid-September says the same. It did come up on the other 9, but every one of those was a question with my name already in it, like "alternatives to Passcite". That's being told, not being found.
What shows up instead: Semrush on 16 of the 30 questions, rankability.com on 20. And the biggest item on my own list isn't a page. It's a Capterra profile: the assistants cite capterra.com on 16 of the 30. Nobody can do that one for me, my own tool included.
That's the first change I'm making. When it's done I'll re-run the same 30 questions and post the result here, whether or not anything moved. My whole product rests on one thing: that changes like this make an assistant change its answer. I haven't proven that yet. One before-and-after won't prove it either, because the assistants drift on their own, but it's more than I have today.
Something I didn't expect: rule 3 is also doing some of the selling I'm bad at. When I email an agency now, there's no pitch. I run one of their clients' sites and send the numbers: 30 questions a patient would type, 22 of them never say the practice's name, and it came up in 2 of those 22. The number does the talking, or it doesn't. I've sent two so far. Ask me in a week.
Rule 2 only half holds this time, though. I'm a customer as a site owner. I'm not an agency, and agencies are who I most want to sell to. That half I'm learning by asking.
So, one question for each side:
If you run an agency: have your clients started asking about ChatGPT or Perplexity yet, or is it still not on their radar?
If you've built something and then stopped using it yourself: what was missing?
The free report needs no signup: https://passcite.com/free-report
Solid lesson. Which channel has worked best for you so far?
Disclosure: we build UtilitySEO, which has a free AEO scan, so there's overlap here.
Your Rule 2 — "I was its customer and I didn't want to open it" — is the most honest product kill signal I've read. A dashboard that confirms what you already fear without telling you what to do next is worse than no dashboard, because it gives the illusion of progress. You're monitoring a problem instead of solving it.
We ran a six-engine AI visibility check on our own site and got a similar gut punch: GPT cited us 5/5 but only from our own pages, Perplexity 4/5, and the list queries where competitors get recommended we were absent entirely. We know the score, we just can't move it yet at DA 3 with ten spam backlinks. Your pivot from "here's your number" to "here's the draft that changes it" is the gap we're still sitting on the wrong side of.
What percentage of users actually approve the AI-drafted changes versus doing it themselves?
The part about not knowing whether you’re building the right thing really hits. AI can speed up shipping, but it can’t replace talking to the right customers and understanding what they’ll actually pay for.
The “number you can’t move is a number you stop looking at” point is really interesting. I think this applies to a lot of SaaS analytics products: it’s easy to give users more visibility without actually giving them more control.
The useful shift seems to be connecting every metric to a possible action and then making it easy to test whether that action actually changed the result. Otherwise, even a very accurate dashboard can eventually become something users check once and forget about.
So many trackers we all have built to never use
Interesting approach. What was the hardest part to get right?
Interesting take. Would you still recommend this approach to someone starting today?
Great breakdown. What feedback have you had from early users?
Really relatable. How much time do you put into this each week?
Clear and practical, thanks. Did anything surprise you along the way?
The “number you can’t move is a number you stop looking at” is a good product test. For an agency workflow, I’d separate diagnosis from the change log: record the exact question set, the cited sources, and the one intervention made for each question. Then rerun the same set on a fixed cadence, while keeping a small untouched sample so normal answer drift doesn’t get mistaken for an improvement.
That distinction between a number you can observe and a lever you can actually pull really resonates. The split between changes your tool can draft and off-site actions like claiming directory profiles feels especially useful; for the before/after run, tagging each question by intervention might help separate assistant drift from changes that moved the needle. Hope the agency conversations give you a clean signal.
this seems interestiing...great work
That part about a dashboard losing its pull the moment it only tells you bad news you already know is spot on.
The shift from just tracking a score to separating automated content drafts from manual things like setting up third-party profiles is what makes a tool actually useful instead of just another tab you stop opening.
Interesting take. Would you still recommend this approach to someone starting today?
The “number you can’t move is a number you stop looking at” insight really stood out to me. It’s easy to build dashboards that measure a problem without actually helping users change the outcome.
I also like the shift from simply tracking AI visibility to identifying why a brand isn’t being surfaced and what actions can actually influence it. The fact that important citations are coming from third-party platforms like Capterra and Healthgrades makes the problem much more interesting — the answer isn’t always “publish more content on your own site.”
The real test will be the before/after results from the same 30 buyer questions. That feedback loop feels much closer to a product people can actually use repeatedly rather than a report they check once.
"That's being told, not being found" is a clean way to put it. I've checked citations for our own stuff and nearly all of them came from GitHub and Reddit threads, not our docs — same shape as your Healthgrades/Zocdoc finding. Curious whether re-running the 30 questions after shipping fixes has moved any of the 21 yet.
Not yet. My G2 profile went up today, Capterra is next, and the re-run comes after both.
Great progress. How did you get your first few users?
Really interesting! What was the biggest challenge you faced while building this?
This is such a good example of the difference between measuring a problem and actually helping users solve it. A visibility score is interesting, but if there’s no clear action behind it, people eventually stop checking it.
I especially liked the shift from “here’s your score” to “here’s what’s missing, why it matters, and what you can do about it.” The feedback loop of analyze → act → re-measure makes the product much more useful.
Really interesting lesson for anyone building analytics or AI tools: a metric becomes valuable when it leads to a decision or action.
The "number you can't move" test is the useful one. Same failure mode shows up in competitive tracking: a weekly dump of competitor release notes looks like progress until nobody acts on it.
What kept me opening those memos was forcing every item into one of three buckets before it could stay on the page: changes our story, changes the roadmap, or trivia. Trivia gets deleted. Everything else gets an owner and a "so what" line.
Curious which of the two agency outreaches you sent came back with a second client's site — that second-client signal feels like the real ruler for whether agencies want a recurring loop vs a one-off report.
Interesting take. Would you still recommend this approach to someone starting today?
The point about a number you can't change is the useful test. Reporting products become guilt dashboards when every bad result stops at the problem. Each one should point to a next action. I'd make the first screen focus on one query. Show the gap, then suggest the smallest change worth testing. Otherwise, users learn something depressing and have no reason to come back.
The bit about letting the number do the talking in outreach is the most interesting thing here. Sending a prospect their own gap data as the pitch removes the entire credibility problem that most agency cold email has. They're not deciding whether to believe your claim, they're looking at their own numbers.
The 22-of-30 stat you're sending is actually a stronger hook than it looks. It's not "here's what you're missing" — it's "here are 22 things patients searched for where no AI mentioned your client's name." That's a loss, not a gap. Losses move people faster than gaps do.
On the agency side: are you leading with the Capterra gap specifically, or the full 22? Because if the biggest item on their list is something only they can do, that's also your strongest proof of concept for paid — if you find it, they still need your help to fix everything else.
The rule about needing a ruler really resonated—an observation that cannot lead to a next action quickly becomes dashboard noise. How are you separating changes caused by your own work from normal model drift when you rerun the same questions?
The move from reporting a visibility score to an action loop is the strongest part here, especially separating what the tool can draft from off-site work like claiming a Capterra or Healthgrades profile. Re-running the same 30 buyer questions is also a great way to keep the feedback honest despite model drift. One useful next layer might be tagging each recommendation by owner and effort, so an agency can turn the findings into a client checklist. I’ll be interested to see what changes in the before/after run.
Interesting take. Would you still recommend this approach to someone starting today?
The point about a number you cannot move becoming a number you stop looking at really resonated with me. I think that applies far beyond AI visibility. Tracking something is easy to confuse with actually improving it.
I also like the distinction between what a tool can help with and what still requires action outside the tool. That feels much closer to how marketing actually works in practice. Curious to see what happens when you rerun the same 30 questions after making those changes.
Interesting take. Would you still recommend this approach to someone starting today?
"A number you can't move is a number you stop looking at" is exactly what happened with our site audit. The first version just gave a score, people looked once and left. What changed it was adding a one-click fix that applies the findings, so the score has somewhere to go.
Also +1 on the off-site part. Google kept autocorrecting our brand name to a similar one until we had profiles on outside sites pointing back to us. Most of that fix wasn't on our own domain at all.
The idea of building an AI visibility tracker and then realizing you no longer use it is an interesting product lesson. It raises a useful question about whether the tool solves a recurring problem or simply provides information that is interesting to check occasionally.
Yeah the I stopped opening it part hit hard. I had the exact same thing with a simple rank tracker I built last year it just sat there telling me the number was bad and I already knew that. After few weeks I never logged in again.
The Healthgrades Capterra angle is the real insight though. Most of these AI visibility tools still act like everything lives on your own domain. Once you see how often the models just pull from those review sites it becomes obvious why a blog post never moves the needle.
This is a really interesting take. I like the point that tracking the score alone doesn’t mean much if you don’t know what to do with the data.
Cool, I tried it
Did anything in it look wrong for your site?
Useful write-up — the visibility vs vanity-metrics split resonates. We're shipping a free recruitment toolkit (Integrity screen + CV parse) with the same "say the limits out loud" approach: https://firstpass.surge.sh — First Pass desk
How does Passcite find positive prompts that customer will query on ChatGPT / Perplexity ?
It reads your site for what you sell and who buys it, then asks what a buyer asks while choosing: who's best, what it costs, how the options compare. Every question is in the report, so you can judge whether a customer would really type it.
Not an agency, but I'm your "site owner" customer this week — doing by hand what Passcite automates.
Yesterday I ran my first GEO baseline: 4 buyer prompts through ChatGPT and Perplexity for my product (PropelAI Studio — proposals, e-signatures and payment-collecting invoices for freelancers). Mentioned in exactly zero answers. Instead the engines recommended PandaDoc, Bonsai, Proposify, Bookipi — and for my Nigerian market, they described a stitched-together workflow: invoice tool + Paystack link + manual reconciliation in a spreadsheet. That manual mess is exactly what my product deletes, and no engine knows it exists.
So this week I did your "How" layer manually: robots.txt + llms.txt + schema deployed, AlternativeTo submitted, comparison page built, directories started. Your point that most of what assistants read isn't on your website at all — that stung. My Capterra attempt was blocked by their signup flow, and your own numbers (16 of 30 citing capterra.com) suggest that's the single biggest lever I'm missing.
I'll re-run my 4 prompts every Tuesday and track movement. If my product starts appearing, that's an outside before-and-after data point for your thesis.
The thing I'd pay for, from a fresh user's view: not the visibility number, but a ranked list of which third-party profiles actually move MY prompts. "Healthgrades, not your blog" is the one sentence a founder can't produce alone.
The free report ranks the profiles you can claim by how many of the questions you lose cite them. Capterra's own signup failed for me too. but you can try g2.
Nice
Your Rule 3 (the ruler) is the most transferable insight in this post. One thing I'd add from watching how these dashboards get used: "mentions" is a lagging metric. What actually moves is citations — assistants cite pages that state a specific, checkable fact (a number, a named mechanism), not pages that carry brand messaging. That's why the "what your own site never says" gap matters so much: AI assistants don't quote adjectives, they quote evidence.
The ruler rule is the hard part. We ran into it from the analytics side: the visits AI assistants do send show up as referrals, so the metric you can move looks like it's working even when the assistant answered without a click. Logging the answer itself, not the visit, was the only version we trusted.
The Capterra result stands out, that's not something you can optimize the way you can a blog post or a page you control. Are you going to try influencing that indirectly, or just treat it as a fixed cost you can't touch?
Neither. I set up the profile myself, so it's direct, just manual. What no tool can do is set it up for me.
The split between what the system drafts (automated) and what only you can decide (manual) is the key insight here. Most measurement tools stop at "here's the problem" - but knowing a problem exists doesn't tell you which solution to pick. The fact that agency clients never show up on Zocdoc, while the tool catches it but a blog post can't fix it, means you're measuring the gap between "what's wrong" and "what can be done." That's where most metrics go dark.
Stopping after you already built the tracker is usually a product-shape problem, not a motivation one.
A score or trend chart answers a question once. What keeps me opening the tool is a short weekly list: which prompt we lost, who got cited instead, and one concrete content fix (FAQ block, comparison table, or source page) with an owner and a due date. Without that handoff, I check once, feel informed, and never come back.
I keep a free AI Answer Visibility Checklist for that prompt-set → evidence → next-fix loop:
https://eastwestkonnex.gumroad.com/l/ai-answer-visibility-checklist
When you stopped using yours, was it because the answer was clear — or because the next action never got written down?
Interesting take. Would you still recommend this approach to someone starting today?
Interesting take. Would you still recommend this approach to someone starting today?
Nice progress. What is the next thing you are focusing on?
What made you pick this stack over the alternatives?
Interesting take. Would you still recommend this approach to someone starting today?
This is useful. How are you finding your first users so far?
Interesting take. Would you still recommend this approach to someone starting today?
Interesting take. Would you still recommend this approach to someone starting today?
Solid lesson. Which channel has worked best for you so far?
Thanks for sharing the numbers, that makes it much easier to follow.
Interesting approach. What was the hardest part to get right?
Interesting. How are you measuring whether it is working?
Really relatable. How much time do you put into this each week?
Good point. Did you test that with users before committing to it?
How did you decide this was worth building in the first place?
Nice progress. What is the next thing you are focusing on?
Nice progress. What is the next thing you are focusing on?
Interesting. How are you measuring whether it is working?
Appreciate the honesty here, most people only share the wins.
Good point. Did you test that with users before committing to it?
Interesting approach. What was the hardest part to get right?
Interesting approach. What was the hardest part to get right?
Good point. Did you test that with users before committing to it?
What made you pick this stack over the alternatives?
Interesting approach. What was the hardest part to get right?
Appreciate the honesty here, most people only share the wins.
What made you pick this stack over the alternatives?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
Interesting. How are you measuring whether it is working?
Interesting approach. What was the hardest part to get right?
Interesting. How are you measuring whether it is working?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
Interesting. How are you measuring whether it is working?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
Good write-up. What would you do differently if you started again?
Appreciate the honesty here, most people only share the wins.
Clear and practical, thanks. Did anything surprise you along the way?
Good point. Did you test that with users before committing to it?
What made you pick this stack over the alternatives?
Interesting. How are you measuring whether it is working?
Good point. Did you test that with users before committing to it?
With two agency outreaches underway, what response would distinguish genuine recurring client demand for AI visibility from curiosity about a new reporting metric?
If they come back with another client's site. One report can be curiosity; a second client means it's becoming part of the work.
That second-client signal is pretty clean. Could be useful to compare notes by email sometime, if you’re open to it.
Interesting. How are you measuring whether it is working?
Interesting approach. What was the hardest part to get right?
Good point. Did you test that with users before committing to it?
Appreciate the honesty here, most people only share the wins.
Good point. Did you test that with users before committing to it?
Good write-up. What would you do differently if you started again?
Clear and practical, thanks. Did anything surprise you along the way?
What made you pick this stack over the alternatives?
Interesting approach. What was the hardest part to get right?
Interesting approach. What was the hardest part to get right?
Appreciate the honesty here, most people only share the wins.
Good write-up. What would you do differently if you started again?
What made you pick this stack over the alternatives?
Clear and practical, thanks. Did anything surprise you along the way?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
Interesting approach. What was the hardest part to get right?
Interesting. How are you measuring whether it is working?
Love this angle. Building Xstream4K right now so this hits close to home — what made you look into it in the first place?
This is great work — reminds me of some of the calls I've had to make building Xstream4K. What would you do differently if you started over?
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Dogfooding is the real filter. The split between buyer questions and what assistants actually surface feels especially useful. What made Passcite worth keeping in your own workflow when the first version failed the open it test?
The split that mattered is a different one: what a tool can draft for me, and what only I can do.
The first version stopped at a number. This one ends in a list I can work through and check again
Interesting approach. What was the hardest part to get right?
What made you pick this stack over the alternatives?
Good write-up. What would you do differently if you started again?
Interesting. How are you measuring whether it is working?
Clear and practical, thanks. Did anything surprise you along the way?
Good point. Did you test that with users before committing to it?
Appreciate the honesty here, most people only share the wins.
Interesting approach. What was the hardest part to get right?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
Interesting. How are you measuring whether it is working?
Good write-up. What would you do differently if you started again?
Interesting approach. What was the hardest part to get right?
Clear and practical, thanks. Did anything surprise you along the way?
What made you pick this stack over the alternatives?
Interesting approach. What was the hardest part to get right?
Appreciate the honesty here, most people only share the wins.
Really good writeup, thanks for sharing it. What's the next thing you're planning to try here?