How do you handle credit card charge disputes?

We have a few thousand CC transactions a year and an occasional dispute (1-3 a year). Given the time it takes to respond to the dispute and the relative small disputed charge, I normally accept it and go on with business but is this the smart thing to do?

For example I'm not sure if this triggers statistics on the payment provider for accepting the dispute instead of getting back and forward with the evidence. This wastes both my and my (ex)customers time even though I have strong evidence.