people consistently overrate their AI skills and its messing up our product decisions

we're building aisa.to — an AI that assesses peoples AI skills through conversation instead of quizzes. and the single most consistent pattern we see is the self-rating gap.

ppl come in rating themselves 7 or 8 out of 10. then we actually walk through how they use AI in their day to day and theyre closer to a 4 or 5. not because theyre bad at it, but because most people confuse "i use chatgpt daily" with "i use AI effectively."

this isnt just an interesting data point — its messing with our product decisions. when we surveyed potential users about what features theyd want, everyone said advanced stuff. multi-agent workflows, custom model tuning, etc. but when we watched them actually work? most needed help with basics like verifying AI output and knowing which tool to use for what.

the gap between what people think they need vs what they actually need is massive. and if youre building anything in the AI tools space you probably have the same problem — your users are telling you they want expert features when they havent nailed the fundamentals yet.

curious if other builders are seeing this too. are your users accurately self-reporting their skill level? or are you designing for an audience that doesnt quite exist yet?