we stopped asking users to rate their AI skills. here's what we do instead

early on we had a pre-assessment survey asking people to rate their AI proficiency 1-10. classic mistake.

the data was useless. people who rated themselves 8/10 would describe workflows that were basically "I paste into ChatGPT and hit enter." people who said 4/10 were running multi-tool setups with verification steps.

so we killed the survey. now aisa.to just has a conversation with you and scores what you actually demonstrate, not what you claim.

the gap between self-reported and observed skill is consistently 2-3 points off. confident users overestimate, cautious users underestimate. almost nobody is calibrated.

biggest lesson: never trust the user to evaluate themselves on the thing you're trying to measure. that's your job, not theirs.