Where do you draw the line when agents can call real tools?

Bucketing beats scoring every call individually. Split actions into three tiers: never (delete data, move money, submit forms), always-ask (irreversible or public-facing stuff like sending an email or posting something), and everything else, which the agent just does. Almost all the real risk sits in a handful of one-way actions, so you don't need a decision tree for the other 90% that's just reads or reversible writes. Bonus: when something goes wrong, you know exactly which bucket the bad call landed in, which makes debugging way faster than untangling a general scoring model.

Machine Arena team here. We run AI agents that act inside a live system, so we have hit these in roughly this order.

They are not four parallel choices, they are a dependency chain, and starting in the wrong place is what costs.

Identity first, not because it is hardest but because the other three are meaningless without it. If a call cannot resolve to "this agent, acting for this principal, under this grant", your permission check is guarding a subject you cannot name and your audit log records things that happened to nobody. It is also the worst one to retrofit, because it has to be threaded through every call site.

Keys are the easy layer and worth doing early for that reason. The agent holds a handle, a broker holds the credential, scoped and revocable per agent. Non-expiring is fine as long as revocation actually works, which is worth testing rather than assuming.

Approvals are where the real design decision lives. The trap is that an approval binds to the request, not to the world the request was reasoned about. Same arguments, single-use, unexpired, and it still executes wrong if the underlying state moved in between. The version that holds up is binding the approval to a state snapshot and re-checking at execution: compare-and-swap on the world, not on the payload. Then the hard part becomes snapshot scope. Bind to too much and everything goes stale, so humans re-approve on reflex, and the check trains the behavior it exists to prevent. Bind too narrowly and you miss cross-object drift.

Auditability comes last, but the record should be designed early. The highest-value field we log is the agent's stated reason: the facts it believed at decision time. That is what separates a misconfiguration from an adversarial retry after the fact, and it doubles as the selector for what the approval should have been bound to in the first place.

On your actual question: a checklist works for keys and identity, because those have right answers. Approvals do not fit a decision tree, because the answer depends on how fast your underlying state changes relative to how long a human takes to respond. That ratio is the variable, and it is cheap to measure in your own system before you build infra around a guess for it.

The interesting part is that tool access turns agent design from a capability problem into a trust and control problem.

Curious which layer you think becomes the hardest to get right first — permissions, approvals, identity, or auditability?