We stopped measuring AI skills like they're exam questions. Here's what we're building instead.
Quick build-in-public update from AISA (aisa.to).
We've been working on something that keeps surprising us: when you measure AI skills through conversation instead of multiple-choice tests, you get completely different results.
The backstory: Most AI skills assessments today work like certification exams. Pick A, B, C, or D. The problem is they measure knowledge recall, not whether someone can actually use AI tools in their work.
So we built a conversational assessment. 20-40 minutes of real dialogue. The AI adapts based on your answers. It doesn't ask "what does RAG stand for" — it asks you to walk through how you'd approach a problem, then follows up based on what you say.
What we're finding:
- People who score high on traditional AI quizzes sometimes struggle to articulate how they'd actually apply the tools
- People who use AI daily but couldn't pass a terminology test often demonstrate strong practical fluency
- The gap between "knowing about AI" and "being skilled with AI" is wider than we expected
We measure across 5 dimensions and 11 criteria, using 10 different personas that adapt to the conversation. It's not a pass/fail — you get a detailed profile of where you're strong and where you're developing.
Why this matters for the IH community: If you're building AI-adjacent products, understanding where your users actually are in their AI journey helps you build better onboarding, better docs, and better products. The "everyone knows how to prompt" assumption is costing builders real money.
Early numbers are still small but the signal is consistent. Happy to share more as we learn.
Would love to hear from other builders — how are you thinking about your users' AI skill levels? Are you designing for power users or beginners? And how do you know which one you're actually getting?
