Building an AI agent that controls your Android phone by voice/text — how should I handle the "trust" problem?
Day 1 building StareBrain — an AI agent for Android that turns
"text Sarah I'm running late" or "add dentist tomorrow 2pm" into
an actual action on your phone, no menu-digging.
The core design problem I keep coming back to
An agent that just does things on your phone is scary. So instead
of "what can it do," my main constraint is "does it show me exactly
what it's about to do, in plain English, before it does it."
Every action needs an explicit confirm. Every permission is granular
and revocable. That trust layer feels like it matters more than the
command set itself.
What I'm building (MVP, kept deliberately narrow)
- Kotlin/Compose Android app ↔ FastAPI backend ↔ LLM (starting with
Nvidia NIM) for intent parsing - 5 command types only: SMS, calendar, device toggles (wifi/bluetooth),
app launch, phone calls - No voice input, no multi-step workflows, no subscriptions — yet
Where I'm at
Architecture's mapped, starting backend (auth + command endpoints +
LLM prompt) this week. Rough goal: 12 weeks to a working beta.
Question for this community
Anyone here built something that acts on behalf of a user
(not just suggests)? Specifically: do you make the user confirm
once per action, or once for a whole multi-step plan? I keep going
back and forth — one confirm feels more trustworthy in the moment,
but it also means approving things you haven't seen play out yet.
The trust problem is clearly central to the product. Curious what early testers consider an acceptable confirmation flow once they can actually use it.
Honestly, no testers yet — still on day one, backend isn't even built. But it's exactly the question I'm trying to answer before I write the confirmation UI, so I'd rather guess right early than redo it later.
My current best guess: confirm once per action for the simple stuff (send this text, add this event), but anything higher-stakes (a call, a multi-step chain) gets its own explicit confirm even if it's part of something already approved. Basically scoping trust to what was actually shown, not blanket-approving whatever comes next.
I'll know a lot more once I've got real people using it — will report back in this thread when I do. If you were building this, where would you draw the line on when a re-confirm is worth the friction?
That makes sense, especially separating simple actions from higher-stakes chains. Since you’re still pre-backend, I’d be more interested in what early testers actually find intuitive than trying to predict the perfect line now.
Fair, and honestly agree — I'll hold the confirm-once-vs-per-step guess loosely rather than build it in stone. Just opened a waitlist today for exactly this reason, so there's a path to actually finding out rather than guessing indefinitely. Appreciate you pushing on it either way.
Makes sense — the waitlist should give you much better signal than guessing at this stage. I’ve enjoyed the exchange; if you’re open to continuing it outside IH, what’s the best email to reach you at?