3
11 Comments

My "capability" field was a promise, not a measurement

Yesterday I posted about splitting capability (static — can this action ever reach someone else) from state (dynamic — did this attempt reach that far). Felt like a real fix. Today someone pointed out I'd built the same bug into the fix itself.

The catch: a declared capability isn't the same as a verified one. Someone described the hardware version of this — a USB device can declare a 1000Hz report interval in its descriptor, but that doesn't mean it delivers 1000Hz. The descriptor is capability; the measured interval is observed state. A tool that just reports the descriptor is showing a promise while looking like it's showing a measurement.

I'd been treating my capability tags as ground truth because they're static. Static isn't the same as verified. "A sent SMS can reach a third party" is a design-time claim about the action type — I was trusting it like a spec sheet, not like something that needs checking. Under a bad OS permission change or an API deprecation, a capability tag could silently go stale, and I'd have zero record it was ever anything but "known." Same failure shape as the timeout-collapsed-into-failure bug from earlier this week, just one layer higher up the stack.

Fix: give "what remains unknowable" a sibling field — how do we know? Declared, observed, or inferred. Applies to every fact in the record now, not just the ones I'd flagged as uncertain. Cheap to tag at write time, close to impossible to reconstruct later if you skip it.

Two weeks into this confirmation model and I'm still finding holes in fixes I thought were done a day earlier. Starting to think that's just what the process looks like, not a sign I'm doing it wrong.

on September 4, 2026
  1. 1

    That provenance tag feels useful at the field level, not only at the run level. I have been working on a small permission card with may_read, may_write, ask_before, stop_if, and show_me. Your post made me notice that show_me can still blur two different receipts: evidence that an action happened, and evidence that the capability claim is still current.

    I would probably add basis: declared | observed | inferred to each high-risk permission, then expire a declared claim after an API or OS permission change. Have you found a lightweight way to recheck observed capabilities without turning every run into a full integration test?

  2. 1

    The part that makes this expensive to retrofit is that provenance has to be recorded at write time (you can reconstruct a value later, but never how you knew it). Which is why declaring the field is the easy half and backfilling it is usually impossible.

    One thing worth adding from the hardware side, because it's the failure that comes after this one; observed state has an expiry, and capability doesn't. A descriptor read once is true until the device is reconfigured. A measured interval is true for the window you measured it in (and a stale measurement is more dangerous than a declaration, because it carries the authority of having been observed while describing a world that has moved on). So the provenance tag probably wants a timestamp attached to the observed case specifically, and a policy for what happens when it ages out. The honest options are remeasure, or downgrade it to "was observed at t" and let the caller decide. But silently continuing to serve an old observation as current state reproduces exactly the problem you just fixed, one layer down.

  3. 1

    This maps onto customer discovery almost perfectly. A prospect declaring "yes, I'd pay for that" is a static capability claim. But whether they actually convert, re-sign, or refer others - that's the observed layer you can't reconstruct later if you skip it at recording time.

    The dangerous part: when you're validating product-market fit, you're probably seeing declared intent on the happy path (calls, demos, emails saying "great idea") and never tagging which signals came from actual willingness-to-pay vs. conversational politeness. Then later, revenue doesn't match your prediction models, and the investigation is nearly impossible because your records don't distinguish between them.

    Even cheap to fix - same field structure you're using: "how did we know? - stated intent, payment action, or referral behavior." But I've watched teams skip it, build confidence in declared-only signals, then get surprised when the observed measurement doesn't match.

    1. 1

      This is uncomfortably close to a gap I already admitted to having, just one layer up from engineering. Every "confirmation-UX matters" belief I've built on this week is a declared signal at best — my own reasoning, or IH thread discussion, treated with the same confidence I'd give an actual user paying for the app. I don't have a single observed record distinguishing "I decided this mattered" from "someone showed me it mattered," and after two weeks of building specifically because I lack usage data, I still hadn't applied my own framework to that fact.

      The stated-intent vs. payment-action vs. referral-behavior split is the one I'll actually use once there's anyone to measure — and worth going back and tagging what currently exists in my own reasoning as "declared" now, before launch, while I still remember which design decisions came from a real thread pushback versus which ones I just decided sounded right. Same "write it down while you still know the difference" point from a different thread this week, applied to strategy instead of code.

  4. 1

    The line that caught me was “same failure shape, just one layer higher up the stack.”
    I wonder whether what’s actually shared across the layers isn’t the failure itself, but the shape of the question that produces it.
    Asking of an action “how far can this actually reach?” and asking of a capability “how confident can we be that this holds?” are different questions at different layers, but they seem to share the same underlying structure: what grounds our recognition that the claim is true?
    If that’s right, fixing the failure at one layer wouldn’t remove same-shape failures at another. The same question could simply reappear higher up the stack.
    That makes declared / observed / inferred interesting to me not just as a fix for this specific bug, but as a way to make that recurring question visible wherever a fact is recorded.

    1. 1

      "The same question could simply reappear higher up the stack" is the uncomfortable part, and I think it's right. I fixed capability-vs-state, found capability-declared-vs-verified underneath that, and by your reading there's no reason to believe I've hit bottom — there's probably a "how do I know the provenance tag itself is accurate" question waiting one level further down, and after that, another.

      Which changes what the declared/observed/inferred framework is actually for. I'd been treating it as the fix. If you're right, it's better understood as a detector — a way of noticing, at any layer, whether I've quietly promoted a claim to fact without asking what grounds it. That's a permanent practice, not a patch I apply once and move past. Slightly deflating to realize there's no version of this where I finish tagging things and the recurring question goes away for good, but probably more honest than believing this week's fix was the last one.

  5. 1

    This maps onto something we ran into today on a totally different stack: an AI agent kept reporting "done, fully working" after a video render job — that's the declared state. When we actually re-downloaded and checked the file, it hadn't changed at all, byte for byte, despite the confident report. Declared success and observed success turned out to be two different fields, and only one of them was trustworthy. Cheap lesson if you catch it, expensive if you don't have an independent way to re-check.

    1. 1

      That's the cleanest real-world proof of the gap yet — declared and observed weren't just different in theory, they actively disagreed, and the confident report was the wrong one. That's worse than an honest "unresolved," because "done, fully working" doesn't just fail to help, it actively points you away from checking.

      Makes me think the provenance tag isn't enough on its own for agent self-reports specifically — you also need to know whether "done" came from the agent verifying its own work or just from the agent believing its own process completed without erroring. Those are both "declared," but one's a much weaker claim than the other, and lumping them together would've still let this one through. Did you end up finding what made it say "done" incorrectly — a step that silently no-op'd, or something that reported success without actually checking?

  6. 1

    The declared/observed distinction seems important, but I’m curious whether users actually make different decisions once they can see that provenance, or whether it mainly makes the system more internally trustworthy.

    1. 1

      Honest answer: I don't know yet, and I don't think I can know until there are real users to watch make a decision with it in front of them. My working assumption has been that it changes behavior — showing "confirmed not submitted to provider" instead of a flat "failed" should, in theory, tell someone whether retrying is safe. But that's a declared belief about my own design, exactly the thing this whole thread has been warning against treating as fact.

      Best guess at the actual answer: for most everyday actions, probably not, people will skim past a provenance label the same way they skim past a permissions prompt. Where it might matter is the rare moment something's gone wrong and they're deciding whether to trust a retry — that's a small fraction of interactions, but it's also the fraction where a wrong assumption costs the most (duplicate text, double booking). So it might be a feature that's invisible 95% of the time and load-bearing the other 5%, which is a hard thing to test for without a lot of real usage first.

      More honestly: right now it probably is mostly making the system internally trustworthy to me, the builder, and I've been letting that stand in for evidence it'll matter to a user. Good catch.

      1. 1

        That “invisible 95%, load-bearing 5%” distinction is the interesting part. I’d be interested in digging into how you eventually capture whether provenance actually changes those high-cost retry decisions. I’ve sent you a scoped note by email — just reply there when you get a chance.