What are your challenges with user interviews?

Doing user interviews is essential to building a product people want. I'm curious, what is the hardest part of that for all of you? Let's discuss.