Don't do that. First, I've done many comments that get a single downvote to 0. Typically, it's a little controversial/against the grain, and it gets the "disagree" downvote. Those are typically voted back up to 1 pretty quickly (I also make a point of upvoting posts that have clearly been disagree-downvoted, even if I wouldn't otherwise have upvoted it).

Second, there's the point of HN karma, especially of the magnitude that can never reach the leaderboard, is completely worthless. I'm sure being on the leaderboard (ie. minor HN royalty) can open some doors, if you want it to, but yours and mine never will.