the hardest product decision we've made: how honest should an AI be about your weaknesses

when we started building aisa.to, we had this naive idea that the assessment would be the hard part. turns out the hardest decision was how direct to be with the feedback.

early version was too gentle. users loved the experience, learned absolutely nothing. someone scores low on verification and the report says 'there's room to develop your evaluation approach.' nobody reads that and thinks 'I have a real gap here.'

so we tried being direct. blunt scoring, here's exactly where you're weak. people felt attacked. didn't come back.

what actually works: evidence-based feedback tied to specific things the person said during the conversation. not 'you're bad at verification' but 'when you described how you use Claude for research, you mentioned accepting the first output without cross-checking — thats a pattern we see in most users and its the single biggest differentiator between casual and advanced AI use.'

when you show someone their own words and explain what those words reveal, they can't argue with it. its not judgment, its their behaviour reflected back with context.

biggest lesson for other builders: people don't want honest feedback or kind feedback. they want evidence they can't dismiss. build the proof into the product