I almost shipped an AI feature that "mostly" worked. That word nearly got me.

I build TextStack, an open-source reader for technical books. It's just me.

A few weeks ago I was adding a thing called "Ask This Book." You're reading, you've got a question, you ask it, the book answers. Simple on paper.

Then one requirement got into my head and wouldn't leave: a reader sitting on chapter 3 should never get spoiled about something from chapter 30.

So I did the obvious thing. I told the model: don't reveal anything from chapters the reader hasn't reached yet. Ran it. And honestly? It worked. Like, 90-something percent of the time it did exactly what I wanted.

I sat there pretty pleased with myself for about ten minutes. Then it hit me.

"Mostly" is useless here. A reader only needs to get spoiled once to feel like the app betrayed them. And the thing that scared me wasn't the bug. It was that I had no way to tell when this feature was actually finished. With normal code something yells at you when you mess up. The build breaks. A test goes red. Here? Nothing. The answers were fluent, confident, well formatted, and occasionally just wrong, and they looked exactly the same as the right ones. In a demo "mostly works" and "done" are literally indistinguishable. I was about thirty seconds from calling it shipped because it felt right when I clicked around.

What got me out of it wasn't a clever prompt. It was forcing myself to write down, in plain words, what failure even meant before I wrote another line:

Spoilers leaked: zero. Not low. Not "good enough." Zero.

And the second I wrote that down the whole approach collapsed, in a good way. Because no prompt can promise you zero. Prompts are a vibe, not a guarantee. People phrase questions in weird ways, models get updated, behavior drifts. You can't get a hard promise out of a soft instruction. Writing the bar down first told me the prompt idea was a dead end before I burned a week polishing it. The actual fix turned out to be boring: stop asking the model to hide future chapters and just never hand them to it in the first place. It can't spoil what it never saw.

I want to be clear I didn't figure this out because I'm smart. I figured it out because I wrote down how I'd know it was broken before I built the thing that could be broken. The definition did the work.

I think this is the part nobody tells you when you start bolting AI onto a product. We all carry this habit from normal software: build it, ship it, write tests later if there's time. For AI features I'm now convinced that's backwards. The "test" is basically the only honest definition of what done even means. If you build first, your test can only grade what you already made. If you write it first, it starts shaping the product, and sometimes it hands you a better design than the one in your head.

And selfishly, as a solo founder, that written-down bar is the only thing that argues back. I don't have a co-founder or a QA person to tell me "hey this is half broken." That one sentence is the closest thing I have to someone saying no.

So now every AI feature in TextStack starts the same way. Before I build anything I write one ugly sentence: how will I actually know this is working? It feels slow on day one and saves me days by day three.

Anyway, curious how the rest of you handle this. When you ship something AI powered, do you actually have a definition of "done," or is it a good demo and a prayer? Genuinely asking, because a month ago I was firmly in the prayer camp.

(If anyone wants the boring technical version, how the retrieval and the spoiler gate actually work, in C#, I wrote that up on my blog: https://vasyl.blog/2026/06/17/evals-before-rag/. TextStack itself is open source, textstack.app.)