Me too. I was pretty miffed the first time I was forced to confront my societal privilege as a straight, white, Christian teen. It felt like I was being picked on or blamed for something I had no control over. (Not Christian any more, haven't been for a long time. I enjoy less privilege now as a woman and a Heathen.)I actually do understand why the OP is thinking the way he is.
White men in particular are often unsettled by the realization that the world is built around them. Not because they're bad, but because they've simply never thought about the world in that way before. And the first time you're made to think about it like that, you feel like a jerk. No one likes to feel that way. But folks generally get over themselves.