turns out AI skill barely transfers across tools — and that changes how we build assessment

one pattern keeps showing up in our assessments at aisa.to — someone who's genuinely impressive with ChatGPT will completely stall the moment you shift the conversation to a different tool or a different type of task.

it's not that they're bad at AI. they've built muscle memory for one tool's interface and quirks, and that doesn't port over the way you'd expect. the underlying thinking skills (verification, prompt iteration, knowing when to push back on output) do transfer. but the confidence and fluency? mostly tool-specific.

this is why we moved away from asking "rate your AI proficiency 1-10" in the assessment. that number means nothing if someone's a 9 with ChatGPT and a 3 with everything else. the conversation format catches this naturally because we can explore different scenarios without it feeling like a test.

still early but this insight is shaping a lot of how we design the next version. curious if other builders have noticed similar patterns — skills that look universal but are actually way more context-dependent than expected.