You can't downvote comments that you reply to. If you downvote a comment and then reply to it, the vote is cancelled, I think.

I still see the downvote arrow on steveklabnik's comment though? So you're saying that clicking it won't have any effect? I don't see any downvote arrow on your comment, because it's a reply to mine. Could you downvote my comment that you replied to?