This whole karma thing is pretty new to me (I had been wondering what that means) but I don't think it's all that bad, and responding, "Why don't you just confront me directly?" misses the point.
Perhaps you are all superior enlightened beings, but I know that when someone confronts me about my behavior, my first response isn't "Oh, I'm sorry, I won't do it again," but, "Why do you think that was wrong?" If you've offended someone or they think you're not contributing in a positive way, having a dialogue about it with them means you can defend your actions, if only in your own mind. Sometimes the end result is that it doesn't make people think about things they could do better or differently, it just makes them think, "This idiot doesn't know what he's talking about."
I feel that the very fact that people seem so bothered by karma, and are so eager to call out those who gave negative scores, is an example of what I'm talking about. If you already feel that you are a positive contributor, why do you care so much? Karma just lets you know that, in someone's eyes, you're coming off a little strong, it's a reminder to watch what you say, because others are. It's not like people aren't going to take your posts seriously because you have bad karma.
It's not a big deal, really...