How much can your AI answer change before it's a bug?
I'm building iPulse AI, an Open Agentic Investment Research Platform, and there's a small reliability test I'd like to run: ask the same question twenty times to the same web-search enabled agent , using exactly the same documents and settings.
Different wording wouldn't worry me. Different revenue figures would. The company didn't change its accounts because I clicked again.
I'd start with a narrow question: what changed in this company's cash flow, and what drove the change? Freeze the documents, model version and date. Run each attempt in a fresh session. Record the figures, sources and main explanation, not just the finished paragraph.
Then separate two problems. Did the system pull different evidence? Or did it reach different conclusions from the same evidence? Those need different fixes. I'd also check the answers against the source, because being wrong twenty times is technically consistent.
I wouldn't demand identical judgment on a genuinely uncertain question. But if the conclusion keeps changing, I'd want that uncertainty visible rather than let one lucky run become the demo.
I haven't run this test yet. For people building AI products: which parts of your answer must stay the same when the inputs haven't changed?
The distinction between evidence drift and conclusion drift seems critical for an investment product. When you run the test, which one would make users trust the system less: different numbers/sources, or the same evidence producing materially different investment conclusions?
I would separate factual invariants from interpretive variance. Revenue figures, dates, and cited source passages should remain stable when the inputs are frozen. The explanation can vary, but it should not quietly reverse the conclusion without identifying the uncertainty that caused it.
It may also be useful to test whether twenty runs produce the same recommended action, since users often care more about whether the system changes their decision than whether the wording changes.