Ran the numbers before deciding how much to automate. 38% of doc work turned out to be low hanging fruit
Im working on DocsKoala. it generates docs from merged github prs. before building out more automation i wanted to actually know how much of the problem was automatable versus how much genuinely needs a human, so i ran a real audit instead of guessing.
pulled 14,891 merged prs across 220 public repos. classified each as customer facing feature, internal, bug fix, or unclear. for the 3,623 customer facing ones, checked whether the pr diff and title alone gave enough to draft a doc, and separately whether it ever got documented at all.
1,386 features, 38.3%, were automatable based on the diff alone, no extra context needed from the dev. 541 features, 14.9%, never got documented period, automatable or not.
cost estimate if a human wrote docs for just the automatable chunk manually, 20-30 min each, comes to 462-693 hours and 34.6k-52k dollars across these 220 repos.
this number is basically why DocsKoala focuses where it does. it drafts automatically for the 38% that's clear cut, and flags the rest for a human instead of pretending it can guess context that isn't in the diff. figured people building similar tools or just curious about the "how automatable is documentation actually" question would find the raw split useful.
The line you drew — draft the 38% automatically, flag the rest for a human — is the actual product here, more than the 38% number is.
Why that split is the right one: automation only holds where the spec is already complete. Your 38% is exactly the slice where the diff and the title ARE the whole spec — nothing else is needed to write the doc. The other 62% isn't "AI can't document this," it's "the spec for this change doesn't exist in the diff yet." The failure mode is not drawing that line at all: hand an agent the 62% with the context missing and it won't stop — it'll invent the missing intent and report the doc as done. It's the documentation version of telling something "go do X" with the success criteria left out and getting back a confident "done!" on work that isn't.
So flag-for-human is really demoting the human to a gate on precisely the un-specced part, which is the correct shape. And I'd read your 14.9%-never-documented as the same problem one step upstream — the intent was never captured anywhere. Prompting for it at merge time is how some of that 62% becomes next year's 38%.
This is nice. It shows that more attention is needed for accuracy
Thanks, and yeah, especially for something that ships without a human double checking every line
Interesting that you quantified it before automating — most people (myself included) jump straight to "automate everything" without measuring where the return is highest. How did you measure that 38%? Was it time spent per task or another metric?
its sufficiency check, not time-spent, for each Pr, does the diff + title alone give enough to draft an accurate doc, or does it need context only the dev has (why a default changed, what edge case a fix covers, internal terms)? binary per pr, which is classified via script and LLM judgment rather than manual review.
The 20-30 min figure is separate i estimated general human write time for the automatable chunk
Flagging the rest for a human instead of pretending it can guess is the right instinct, and it is rarer than it should be. I built the same principle into FounderFlow, it grades its own confidence instead of presenting a guess as a fact. Founders trust these tools a lot more once they can see where the tool itself is unsure.
The finding that 38% is automatable from the diff alone is the more useful number here than the aggregate automation rate, because it tells you where the ceiling actually is. I've hit the same wall building on top of PR diffs, a diff shows you what changed but not always why, and the why is usually what a real doc or a real review comment needs. I would chase the 14.9% that's never documented at all before chasing a higher percentage on the automatable slice, since that gap is pure signal that nobody is even trying to answer, not a hard technical ceiling.
I like the name DocsKoala, and I think all companies should run a test similar to yours. Do you have any other projects planned? If so, which ones?
currently, im not specifically planning anything apart from DocsKoala itself. and thanks for the kind words about the name !
The 38% figure feels believable, and calling out the rest instead of making up context is where trust comes from. That missing 14.9% is probably more than a documentation problem. The reason for a change often lives in Slack, on a call, or just in one person's head. I'd try a one-sentence intent prompt at merge time, then let the draft use that with the diff. You can automate the first draft without pretending the diff knows why the change happened.
agreed, and the one sentence intent prompt at merge time is basically what "/koala" tagged commits are for right now, but its opt-in and i dont think most devs would bother with it
Running the audit first before deciding what to automate is exactly the right call, and the 14.9% that never got documented at all is the more interesting number to me. That's not an automation problem, that's a workflow problem. Those features are invisible to users no matter how good the tooling is. The 38% automatable slice is a clean wedge though. Scoping to what's clear cut and flagging the rest is more honest than most doc tools which pretend the ambiguous 62% doesn't exist. How are you positioning this to dev teams who've already got a process (however broken) versus teams who have nothing at all?
that would require two different pitches, but, its probably easy to sell to teams who have nothing because any docs would obviously beat no docs.
however, teams with (broken) tools could be tricker. the pitch isnt to "replace your process", its acutally, DocsKoala could automate that process while still having a human on loop for checking the docs before approving so the human attention could be directed towards something which actually needs human attention.
havent tested that though, feel free to give it a go yourself, id love your recommendations.
The number I'd chase isn't the 38% automatable, it's the 541 that never got documented at all. Docs you'd have written anyway are a time save; docs that would never have existed are net-new coverage, and that's a far easier thing to sell. How much of that 541 overlaps with the automatable 38%? Also worth a caveat: this is public repos, where PR titles tend to be tidier than internal ones, so the automatable share on private repos with 'fix stuff' messages might look different.
havent oulled the overlap yet. i need to check how much of the 541 undocumented also falls inside the automatable 38%, If it happens to be high, thats a much better pitch than the time save framing.
That overlap is the whole ballgame, so split the 541 into two buckets before you run it. The ones inside the automatable 38% are your headline: docs that would never have existed, draftable with zero extra dev input. Basically free coverage. The ones outside it (undocumented, and not automatable from the diff alone) are the harder story — those are exactly what a merge-time intent prompt has to catch, and they're the changes nobody bothered to write up the first time round. If that outside slice is big, the intent prompt stops being a nice-to-have and turns load-bearing. Curious which way it splits once you pull it.
Thats nice!!! A 38% is absolutely wild
Yeah, it is.
I think the most valuable part of this isn't the 38%.
It's that you identified the boundary where automation stops being trustworthy. A lot of AI products try to maximize automation. Defining where a human should take over often creates more confidence than squeezing out a few extra percentage points.
i appreciate that. i went in wanting to push the 38% higher, but the audit convinced me the boundary matters more. and every draft still goes through human review before it publishes, so it's not only "AI ships stuff," but "AI drafts what it's confident about, a human checks it, and the hard 62% gets flagged instead of being guessed
I like where your thinking landed.
Your reply raised a question for me about how you'll decide where that boundary should move over time. I'd rather understand that in the context of your product before offering a conclusion.
What's the best email to reach you on?
i appreciate that and im glad to go deeper. you can contact me at [email protected]
Hi, just a heads-up—I tried sending the note, but it bounced because the contact address appears to be a restricted Google Group rather than a normal inbox. If you're still interested, what's the best email address to reach you on?