I had this idea before as well and I agree with it. It has its flaws but I like it. The second possibility I found was exactly opposite - making it fully public, so everyone can see why and who negrepped someone. In this way, you could actually get an opinion about a given member (let's say, seeing he's got negrep mainly from people you don't respect and they gave it out without a real reason, while positive rep from people you like). The problem of this solution is obvious, everyone can see what you wrote and thus dislike and negrep you just due to your comment.

But I believe we're mature enough not to do such things, aren't we?