I have to agree. Along with what you said about wanting to review a game or having something else to do, sometimes I simply don't want to play the same player again.
It bugs me how players will call you a coward for not rematching them, but I try to just let it roll off. Don't let it get you down!
What is it with people who feel like you 'owe' them a rematch if they lose? Sometimes I'm in the mood to play consecutive games, other times I want to review the game I just played, and other times I might just feel like taking a break, or, heaven forbid, I might actually have something else to do besides play chess. In any case chess is competitive game and it is likely that someone will win and someone will lose. If you can't handle that you might want to find a different hobby. That being said, I like to think of chess players as being gentlemen and good sportsmen, at least publicly. Keep the trash talking and whining to yourself or your group of friends, if that's what you do. Thank you!