My "capability" field was a promise, not a measurement
Yesterday I posted about splitting capability (static — can this action ever reach someone else) from state (dynamic — did this attempt reach that far). Felt like a real fix. Today someone pointed out I'd built the same bug into the fix itself.
The catch: a declared capability isn't the same as a verified one. Someone described the hardware version of this — a USB device can declare a 1000Hz report interval in its descriptor, but that doesn't mean it delivers 1000Hz. The descriptor is capability; the measured interval is observed state. A tool that just reports the descriptor is showing a promise while looking like it's showing a measurement.
I'd been treating my capability tags as ground truth because they're static. Static isn't the same as verified. "A sent SMS can reach a third party" is a design-time claim about the action type — I was trusting it like a spec sheet, not like something that needs checking. Under a bad OS permission change or an API deprecation, a capability tag could silently go stale, and I'd have zero record it was ever anything but "known." Same failure shape as the timeout-collapsed-into-failure bug from earlier this week, just one layer higher up the stack.
Fix: give "what remains unknowable" a sibling field — how do we know? Declared, observed, or inferred. Applies to every fact in the record now, not just the ones I'd flagged as uncertain. Cheap to tag at write time, close to impossible to reconstruct later if you skip it.
Two weeks into this confirmation model and I'm still finding holes in fixes I thought were done a day earlier. Starting to think that's just what the process looks like, not a sign I'm doing it wrong.
"Cheap to tag at write time, close to impossible to reconstruct later if you skip it." True, and write time is not the only place the tag dies. You already moved from calling the field a fix to calling it a detector, so this is about where that detector has to survive.
I mark unverified things in my notes with square brackets so a later session sees what was never checked. One long session got auto summarized. The summary kept the value, a fee rate I had assumed and never confirmed, and dropped the brackets. It came back as a flat statement of fact, and the next session read it that way. The model never lied. The compression did, by discarding the one piece of metadata that said this is not known. Honest limits: it happened once in the wild, I could not reproduce it on demand, the brackets were not a fix since they are the thing that failed, and my own experiment is at $0 in sales.
So there are two ways to lose provenance. Yours is the tag you skip at write time. The other is the tag you did write, correctly, into a layer that something downstream is allowed to rewrite. A "declared" mark living in a conversation, or in anything that gets summarized, serialized, or handed to another agent, is a claim about how you know with no record of how it survived. I cannot stop a compression step from dropping it, only say where it has to live to be out of reach.
The useful part: this is testable now, without users. Take an already tagged record, push it through whatever compresses or hands off your context, and check whether the tag arrives attached to the same value. Aging out an observation, which Andrewed raised, assumes the tag is still there to age.
That is settleable before launch: either provenance lives in the durable record and one round trip proves it, or it lives in the transient layer, where the field is decoration for exactly the case it exists for.
"A claim about how you know with no record of how it survived" is the exact gap, and worse than the write-time gap because it's invisible until something downstream actually mangles it — you can audit whether you tagged something, you can't easily audit whether the tag will still be there after every future transformation you haven't built yet.
The round-trip test is the part I'm actually going to run, not just admire. My provenance tags currently live inside the same event record as the value itself, not a separate durable store — so the real question is whether every place that record gets serialized, cached, or handed to another part of the system preserves the field or silently drops it because some intermediate step only cared about the value. I don't know the answer, and "I assume the schema carries it everywhere" is precisely the declared-not-observed claim this whole thread has been warning against.
Appreciate the honesty about the failure being unreproducible and the experiment being at $0 in sales — that's a harder thing to admit than the finding itself, and it's exactly the kind of provenance you're describing applied to your own comment: here's what I know, here's what I don't, here's what I couldn't verify. Going to go run the round-trip test today and report back the real result either way.
That provenance tag feels useful at the field level, not only at the run level. I have been working on a small permission card with
may_read,may_write,ask_before,stop_if, andshow_me. Your post made me notice thatshow_mecan still blur two different receipts: evidence that an action happened, and evidence that the capability claim is still current.I would probably add
basis: declared | observed | inferredto each high-risk permission, then expire a declared claim after an API or OS permission change. Have you found a lightweight way to recheck observed capabilities without turning every run into a full integration test?Honestly, no, I don't have a lightweight version of this yet — my instinct so far has been "trust the declared capability until something fails," which is exactly the trap this whole thread is about. A full integration test on every run is obviously too expensive, but "never recheck" is the other failure mode, just quieter.
Best half-answer I have: recheck triggered by event, not by schedule. Instead of polling "is this capability still valid" on every run, invalidate the specific declared tag only when something adjacent to it changes — an OS permission-grant callback fires, an API returns a deprecation header, a provider starts rejecting a call shape it used to accept. That turns it from "verify constantly" into "verify only when there's a reason to suspect drift," which is cheap because most runs won't trigger anything.
Downside I can already see: that only catches drift the system has a hook for. Silent drift — something changes and nothing fires an event you're listening for — still falls through, and you're back to declared-and-stale with no signal. Might genuinely need both: event-triggered invalidation as the cheap common case, plus some slow periodic full recheck as a backstop for the drift you didn't think to hook.
Your basis field on the permission card is the right home for this regardless of how the rechecking gets solved — at least the staleness is visible even before the mechanism to fix it exists.
The part that makes this expensive to retrofit is that provenance has to be recorded at write time (you can reconstruct a value later, but never how you knew it). Which is why declaring the field is the easy half and backfilling it is usually impossible.
One thing worth adding from the hardware side, because it's the failure that comes after this one; observed state has an expiry, and capability doesn't. A descriptor read once is true until the device is reconfigured. A measured interval is true for the window you measured it in (and a stale measurement is more dangerous than a declaration, because it carries the authority of having been observed while describing a world that has moved on). So the provenance tag probably wants a timestamp attached to the observed case specifically, and a policy for what happens when it ages out. The honest options are remeasure, or downgrade it to "was observed at t" and let the caller decide. But silently continuing to serve an old observation as current state reproduces exactly the problem you just fixed, one layer down.
"A stale measurement is more dangerous than a declaration, because it carries the authority of having been observed while describing a world that has moved on" is the line that reorders my priority list. I'd been treating DECLARED as the weaker, riskier state and OBSERVED as the safe one, once verified, always trustworthy. You're right that's backwards for anything time-sensitive — a declaration at least announces its own uncertainty, a stale observation actively hides it behind confidence it no longer deserves.
This also answers something mehmetkocabas asked me a few replies up, about rechecking observed capabilities cheaply — sounds like the answer isn't avoiding the cost of rechecking, it's making staleness itself visible so the caller can decide, rather than trying to solve staleness away entirely. "Was observed at t" with a hard downgrade policy past some threshold is honest in a way that indefinite trust in a single observation never was, even if the observation was correct when it was taken.
Concretely, going to add expiry as a required field alongside basis, not optional: every OBSERVED entry gets a timestamp and a policy for what it downgrades to once stale, instead of quietly remaining "observed" forever just because nothing re-checked it. Matches the round-trip-survival test from earlier in this thread too — an expired observation that still reads as current is the exact same failure as a dropped provenance tag, just caused by time instead of a bad serialization step.
Making expiry a required field is the right call, and I'd go one step further, because tjgarage's point above is the stronger version of mine. The tag doesn't only have to survive write time, it has to survive every copy. The hardware version of that is a spec sheet. Someone measures a report interval on one unit, one afternoon; the number gets copied into a database; and by the second copy it reads as a declaration. The "measured, once, on this unit" part is gone, and nothing about the number itself says it was ever observed. It didn't age out, it got promoted to a fact by being transcribed.
So the test I'd add to the roundtrip one is to push an OBSERVED entry through whatever path copies or summarises it, and check whether what comes out still says "observed at t" or has quietly become bare. If the tag can't survive a summary, the expiry policy never gets a chance to run, which is tjgarage's point exactly. The fix is probably that the value and its provenance have to be one object that can't be split, rather than two fields a summariser is free to keep one of.
This maps onto customer discovery almost perfectly. A prospect declaring "yes, I'd pay for that" is a static capability claim. But whether they actually convert, re-sign, or refer others - that's the observed layer you can't reconstruct later if you skip it at recording time.
The dangerous part: when you're validating product-market fit, you're probably seeing declared intent on the happy path (calls, demos, emails saying "great idea") and never tagging which signals came from actual willingness-to-pay vs. conversational politeness. Then later, revenue doesn't match your prediction models, and the investigation is nearly impossible because your records don't distinguish between them.
Even cheap to fix - same field structure you're using: "how did we know? - stated intent, payment action, or referral behavior." But I've watched teams skip it, build confidence in declared-only signals, then get surprised when the observed measurement doesn't match.
This is uncomfortably close to a gap I already admitted to having, just one layer up from engineering. Every "confirmation-UX matters" belief I've built on this week is a declared signal at best — my own reasoning, or IH thread discussion, treated with the same confidence I'd give an actual user paying for the app. I don't have a single observed record distinguishing "I decided this mattered" from "someone showed me it mattered," and after two weeks of building specifically because I lack usage data, I still hadn't applied my own framework to that fact.
The stated-intent vs. payment-action vs. referral-behavior split is the one I'll actually use once there's anyone to measure — and worth going back and tagging what currently exists in my own reasoning as "declared" now, before launch, while I still remember which design decisions came from a real thread pushback versus which ones I just decided sounded right. Same "write it down while you still know the difference" point from a different thread this week, applied to strategy instead of code.
The line that caught me was “same failure shape, just one layer higher up the stack.”
I wonder whether what’s actually shared across the layers isn’t the failure itself, but the shape of the question that produces it.
Asking of an action “how far can this actually reach?” and asking of a capability “how confident can we be that this holds?” are different questions at different layers, but they seem to share the same underlying structure: what grounds our recognition that the claim is true?
If that’s right, fixing the failure at one layer wouldn’t remove same-shape failures at another. The same question could simply reappear higher up the stack.
That makes declared / observed / inferred interesting to me not just as a fix for this specific bug, but as a way to make that recurring question visible wherever a fact is recorded.
"The same question could simply reappear higher up the stack" is the uncomfortable part, and I think it's right. I fixed capability-vs-state, found capability-declared-vs-verified underneath that, and by your reading there's no reason to believe I've hit bottom — there's probably a "how do I know the provenance tag itself is accurate" question waiting one level further down, and after that, another.
Which changes what the declared/observed/inferred framework is actually for. I'd been treating it as the fix. If you're right, it's better understood as a detector — a way of noticing, at any layer, whether I've quietly promoted a claim to fact without asking what grounds it. That's a permanent practice, not a patch I apply once and move past. Slightly deflating to realize there's no version of this where I finish tagging things and the recurring question goes away for good, but probably more honest than believing this week's fix was the last one.
This maps onto something we ran into today on a totally different stack: an AI agent kept reporting "done, fully working" after a video render job — that's the declared state. When we actually re-downloaded and checked the file, it hadn't changed at all, byte for byte, despite the confident report. Declared success and observed success turned out to be two different fields, and only one of them was trustworthy. Cheap lesson if you catch it, expensive if you don't have an independent way to re-check.
That's the cleanest real-world proof of the gap yet — declared and observed weren't just different in theory, they actively disagreed, and the confident report was the wrong one. That's worse than an honest "unresolved," because "done, fully working" doesn't just fail to help, it actively points you away from checking.
Makes me think the provenance tag isn't enough on its own for agent self-reports specifically — you also need to know whether "done" came from the agent verifying its own work or just from the agent believing its own process completed without erroring. Those are both "declared," but one's a much weaker claim than the other, and lumping them together would've still let this one through. Did you end up finding what made it say "done" incorrectly — a step that silently no-op'd, or something that reported success without actually checking?
The declared/observed distinction seems important, but I’m curious whether users actually make different decisions once they can see that provenance, or whether it mainly makes the system more internally trustworthy.
Honest answer: I don't know yet, and I don't think I can know until there are real users to watch make a decision with it in front of them. My working assumption has been that it changes behavior — showing "confirmed not submitted to provider" instead of a flat "failed" should, in theory, tell someone whether retrying is safe. But that's a declared belief about my own design, exactly the thing this whole thread has been warning against treating as fact.
Best guess at the actual answer: for most everyday actions, probably not, people will skim past a provenance label the same way they skim past a permissions prompt. Where it might matter is the rare moment something's gone wrong and they're deciding whether to trust a retry — that's a small fraction of interactions, but it's also the fraction where a wrong assumption costs the most (duplicate text, double booking). So it might be a feature that's invisible 95% of the time and load-bearing the other 5%, which is a hard thing to test for without a lot of real usage first.
More honestly: right now it probably is mostly making the system internally trustworthy to me, the builder, and I've been letting that stand in for evidence it'll matter to a user. Good catch.
That “invisible 95%, load-bearing 5%” distinction is the interesting part. I’d be interested in digging into how you eventually capture whether provenance actually changes those high-cost retry decisions. I’ve sent you a scoped note by email — just reply there when you get a chance.
Got it, will take a look and reply there.