The number that made us rethink our entire positioning
We built AISA (aisa.to) as an AI skills assessment. After ~1,900 assessments, one stat changed everything: the correlation between self-reported AI confidence and measured ability is r = 0.07 to 0.24. Basically zero.
People who rate themselves 8/10 on AI skills often score below 50/100 on task-based measures. Average across all users: 48/100. 67% fall below "Proficient."
We started thinking we were building a learning tool. Turns out the real value is the gap itself. Organisations do not know where their teams stand on AI capability, and self-assessments are nearly worthless.
The EU AI Act made this concrete. Article 4 requires "sufficient AI literacy" for anyone deploying AI systems. When we published our compliance checklist (aisa.to/blog), compliance teams reached out because they had no way to measure what Article 4 demands.
That shifted our whole go-to-market. We stopped saying "upskill your team" and started saying "measure what you cannot see." Same product, different conversion entirely.
Curious if anyone else has had a regulation or external event completely reframe how they position their product.
The gap is striking. Have you broken the scores out by job role or by how often people actually use AI at work? A company-wide average could hide a team that needs much more support, and that breakdown might be the most useful part of the report for a buyer.
This is a great example of why the problem you discover can be more valuable than the problem you originally set out to solve. The shift from “help people improve” to “help organizations measure the gap” completely changes the buyer and the urgency. I’d be curious how you’re now identifying and reaching those compliance teams.
This is a textbook case of the benchmark becoming the product. The assessment itself is fungible — anyone can write task-based questions — but 1,900 scored assessments is a norms table nobody else has, and that is your actual moat. One caution on the Article 4 angle: compliance buyers purchase on a different clock than learning buyers. The budget sits with legal, the decision is risk-driven, and it is often a one-off audit unless you attach it to a recurring requirement like annual retesting or onboarding screening. If you can make the retest the default — "your Article 4 evidence expires in 12 months" — the regulation stops being just a positioning frame and starts being a retention engine.
Great example of letting the data pick the positioning. That gap between what people say and what they can do shows up in every self-reported survey, which is why task-based measures (or follow-up questions that ask for a concrete example) are so much more useful. The EU AI Act angle is a strong wedge.
r = 0.07 to 0.24 is a great headline, and "measure what you cannot see" is a much sharper promise than "upskill your team". It also explains why the old pitch was hard: nobody buys training for a gap they don't believe they have.
Two thoughts on the Article 4 angle:
Compliance buyers need evidence they can file. A per-team report with the date, the method, and the score distribution, in a format that drops straight into an audit folder, is probably worth more to them than any individual score. That's also the thing they'd forward internally, which makes it your sales asset.
The retest is your retention loop. If the gap is the value, the second assessment three months later showing the gap closing is what renews the contract. I'd build the "before and after" view before anything else on the roadmap.
One question: how do you keep a task-based test meaningful the second time someone takes it? If people remember the tasks, the improvement you show at renewal could be partly memory.
r = 0.07 to 0.24 is a wild number to build a pitch around, and it makes sense that "measure what you cannot see" converts better. we saw a tiny version of this with testers who said the app was fine and then never opened it again, so what people said and what they did barely lined up. did the compliance angle change your pricing too or mostly the messaging?
Really good writeup, thanks for sharing it. What's the next thing you're planning to try here?