I've interviewed 1800 people about how they use AI
Well to be perfectly honest, I built an agent that did that for me (https://aisa.to) but the result is the same. After 1800 interviews we have gathered A LOT of insights, here are my favourites.
-
The least AI-fluent professionals overestimated their score by 40 points. The most fluent underestimated by 27. 67 point Dunning-Kruger gap.
-
Product managers outscore engineers on AI fluency (59.2 vs 53.7). Applied judgement beats technical knowledge.
-
HR people (ironically who make hiring decisions using AI) understand AI the least, with the lowest AI fluency score.
-
People consistently say they are good with AI, but 2 out 3 fail to reach even proficient level.
-
The average AI fluency score across 1,800 professionals is 48 — squarely in the Developing tier. Most people use AI regularly but without systematic practice.
-
Company-wide AI training assumes everyone starts from the same place. The data says starting points vary by 5x within the same team.
-
Same company, same tools, same training budget. The gap between the least and most AI-fluent employee in one team was 82 points (15 to 97).
If you are sceptical (which is good) you can read the full report in the AISA AI fluency index and even take the test for free.
The gap between self-reported AI fluency and assessed fluency is the most interesting finding to me, especially the 82-point spread within the same team. How did you operationalize and validate “AI fluency”—prompting technique, task outcomes, judgment about when to use AI, or a combination? I’d also be curious whether you accounted for self-selection and differences in role or tool access when comparing PMs, engineers, and HR.
We have a very thorough walkthrough of our full methodology if you are curious. aisa.to/methodology that goes into details of what we measure, how, and why these are the most valid metrics to look out for as supported by DOL and Anthropic in their AI fluency reports.
The gap we have is that it's a self selected sample. As opposed to a scientific study where the sample would need to be representative.
On your last question, we expect different things from different roles in terms of tool use and tool access but what we measure is role agnostic and that's why software developers who know how to code don't necessarily automatically score high and an eager PM can be in a better spot when it comes to full on productivity and fluency with today's AI.
Thanks, that clarifies the scope. The self-selected sample is an important caveat: it limits population-level claims, but the role-agnostic scoring can still make within-sample comparisons useful. The PM-versus-developer point is especially interesting—it suggests workflow judgment and willingness to experiment may matter more than coding ability alone. I’ll take a look at the methodology.