we stopped asking users to rate their AI skills. here's what we do instead

early on we had a pre-assessment survey asking people to rate their AI proficiency 1-10. classic mistake.

the data was useless. people who rated themselves 8/10 would then describe workflows that were basically "I paste things into ChatGPT and hit enter." meanwhile people who said 4/10 were running multi-tool setups with verification steps we hadn't even thought to assess.

so we killed the survey entirely. now aisa.to just has a conversation with you and scores what you actually demonstrate, not what you claim.

the uncomfortable part: the gap between self-reported and observed skill is consistently 2-3 points off. confident users overestimate, cautious users underestimate. almost nobody is calibrated.

biggest lesson for any builder in the assessment/skills space: never trust the user to evaluate themselves on the thing you're trying to measure. that's literally your job, not theirs