I get somewhat annoyed when a comment of mine is downvoted but I think you're taking it too strongly. Sometimes people don't like valid comments. And especially since votes fluctuate, don't worry about an initial downvote.

And I don't really agree with "down-voting someone is like flagging them". I only flag the rare completely indefensible troll/spambot comments.