What 100B+ Claude tokens actually look like inside a tiny company

16.9B tokens in 30 days is a number I had to read twice. The 63,996 model turns make it concrete—you're not prompting, you're reviewing an assembly line.

The "confidently wrong implementation" detail resonates. I've had mornings where I undo what an agent built overnight. That trade-off between flexibility and the always-running system is real.

How do you keep yourself from checking in on the system during family time? Is that a discipline thing, or have you found a specific boundary that works?

The "harder to switch off" trade-off matches my experience. Once agents can run overnight, the bottleneck becomes morning triage: reading diffs to decide what to keep and what to unwind. One thing that helped us was having agents write a short decision log into the repo as they work, so the morning review is a page of notes instead of a raw transcript. How do you keep the unwinding cheap at your volume, small commits or just reading everything?

someone upthread asked whether automated checks catch the bad outputs before a human sees them. we tried exactly that and measured it, so here are the numbers instead of my impression of them.

the setup is a pipeline that reads public communities every day and pulls labelled evidence out of them. the obvious next thing to delegate was QA: second model reviews the first model's labels and flags the wrong ones. when we finally calibrated it against 42 rows a human had already ruled on, agreement was 69%. a function that ignores the input entirely and answers "correct" every time would have scored 71.4%. the reviewer was worse than a constant.

it wasn't uniformly bad though. sorted by the reviewer's own stated confidence, its top bucket was 6 for 6 and the one below it fell to roughly 30%. small n and i won't pretend otherwise, but the shape was clear enough to build on: it can raise a hand, it can never clear a row. a "looks fine" from something running at 30% is worse than no signal, because you stop looking.

the less flattering half of that: of the 13 disagreements, 8 landed on the same label, and the cause turned out to be that our own spec defined that label three different ways in three different places. the model wasn't confused, we were. i'd guess a fair share of what gets filed as agent unreliability is really that, and the only way we found it was hand-judging rows.

so on your actual question — what stays human is anything that decides. scoring and state transitions here are plain deterministic code, and the model only reads, extracts and matches. not because a model couldn't do the scoring, but because when a number looks wrong six weeks later i need a line i can point at rather than a prompt i can re-run.

Really interesting to see how a small company can use AI at this scale. Do you think this level of automation creates bigger opportunities for non-technical founders too?

The shift from manual execution to agent-driven workflows is really interesting, especially at this scale.
It shows how AI can handle context gathering and execution while humans stay focused on judgment and strategy.
A similar automation mindset can even help with tasks like a Pinterest Video Downloader, where repetitive work can be streamlined efficiently.
The key is keeping the right human-in-the-loop checkpoints to catch confidently wrong outputs.

The shift from doing every task manually to focusing more on context, judgment, and direction really resonates. I think the same approach can work beyond software development too, especially for researching and evaluating online platforms like digital earning opportunities. The key seems to be having agents handle the repetitive research while humans still verify the important decisions.

The reading-and-editing share being your biggest category is the part I'd dig into, because it's the one line item that's mostly avoidable.

Most of it isn't reading, it's re-reading. A session rediscovers the same five facts about a codebase that three previous sessions already established, because the conclusions lived in a transcript and the transcript is gone. An index makes each lookup cheaper but doesn't reduce how many happen.

Two things that actually moved it for me. Searching is delegated to sub-agents, so the file dumps and the dead ends stay in the sub-agent and only the conclusion comes back to the main thread. And every decision, gotcha and current-state note gets written to a file in the repo the moment it's reached, so the next session opens with the answer instead of deriving it. The rule I hold myself to is that the conversation is disposable and the written record is the real one, and if something only exists in the conversation it isn't finished.

On your actual question: what stayed human longest for me isn't a category of decision, it's anything where no independent check exists. Where a separate tool can say yes or no, I let it run. Where the only available verdict is the agent's own summary of its own work, I read it myself, because a confident summary and a correct one are indistinguishable from the inside.

That’s a pretty interesting shift in how a small team can operate. The part that stands out to me isn’t even the 100B+ token number, but how little of the usage is direct messaging compared with file work, browser automation, and terminal commands.

It also highlights the less glamorous side of agentic workflows. Getting an agent to complete a task is one thing; knowing when its output is actually trustworthy still requires human judgment. I imagine deciding what not to delegate becomes just as important as deciding what to delegate.

Curious how you handle review at that scale—do you have automated checks catching most of the bad outputs before they reach a human, or is the human review still fairly hands-on?

The 'confidently wrong implementation' line is the one that took me longest to handle gracefully. Agents are unusually good at being wrong with high confidence, and the failure mode isn't usually random — it's that they optimized for something that sounded like your intent but wasn't. The signal that matters isn't output quality, it's how far wrong they get before anything surfaces it.

For us, the things that stayed human-in-the-loop longest were anything where the cost of being confidently wrong was asymmetric: customer-facing messaging, any decision that required saying no to something that looked good on paper, and anything where being wrong meant spending two weeks going the right direction for the wrong reasons.

The shift you describe — direction + context + review — is right, but I'd add one more: pattern recognition on the error classes. After you've seen the same type of wrong implementation a few times, you can catch it faster at the context stage rather than the review stage. That feedback loop changes the economics.

The system looks great I really appreciate the work that's gone into it Jan.

We're entering an era where AI agents are becoming the new workforce, handling tasks autonomously while keeping humans in the loop to review, validate, and ensure quality. Businesses are rapidly adopting AI agents, specialized skills, and MCPs to automate workflows, significantly reducing manual effort and cutting task completion time by as much as 90%. This shift is transforming how work gets done, allowing teams to focus on higher-value decision making instead of repetitive operations.

The "tries to" qualifier is the part most agent demos leave out. Once agents run overnight, the bottleneck isn't producing more work. It's proving that something changed in the outside world. The most reliable pattern I've found is to define an explicit output contract, then run a separate verification step. If the evidence is missing, the run should fail. Otherwise, a confidently wrong result can sit there looking like progress until morning.

The token count is a vanity metric, the real story is the shift you describe: direction, context, review, correcting course. At SocialPost.ai our AI PM spends most of her time doing exactly that, catching wrong turns rather than writing prompts. The one place we still keep a hard human gate is anything that touches a customer's published content or billing, agents draft it, a person approves the send.

Really impressive setup, Jan. Congrats on building this with such a small team, and thanks for sharing the real trade-offs instead of just the upside.

I think the biggest shift here is that the bottleneck moves from execution to judgment. Research, drafting, data movement, and repetitive tasks can be heavily delegated, but human checkpoints still matter around irreversible actions and customer-facing decisions.

At this scale, designing the right review system probably becomes just as important as the agents themselves.

Most forget when you yourself build you do a thousand checks. Why should it be different with AI? Plan, build, check, recheck, test, rinse and repeat.

I see reading and editing is your main token usage. I build a little tool for my repo's that makes a database with all references so AI can find things quick without going through thousands of lines. It cut my token usage for reading significantly. Might be usefull.

Mostly delegated the drafting grind - first-pass research, structuring, turning it into an actual finished deliverable. Kept myself in the loop on anything that goes out publicly or touches a live account: final review before publishing, pricing/positioning calls, that kind of thing. The "confidently wrong" risk you mention is real, especially the first few times you let it run further than you're used to - worth checking its output against reality before it snowballs.

The shift from doing the work to reviewing and correcting agent output seems like the more interesting change here.

I’m curious where you’ve found the boundary is hardest to define — which decisions can safely run without you, and which still require human judgment?

The shift from doing to reviewing and correcting is the real change. Most people underestimate how much judgment is still needed — agents multiply your output but they don't replace the decisions about what's worth building.