I mean it was mostly a joke but it’s also semi serious. You seem capable of disagreement without it bothering you much so I don’t get why the number next to a comment is different. Yeah some people use it in a dumb way but that’s true of pretty much everything.
You’re obviously allowed to care and to express that. I hope you don’t hear me saying that I think you’re dumb or anything like that. In fact it’s pretty much the exact opposite. Most of the time when I see people complain about downvotes they’re trolls upset that it hampers their ability to be a troll but you seem like a reasonable person so I am simply seeking to better understand your perspective.
To put it another way I agree with your first sentence but I don’t generally feel like visible downvotes prevents that from happening. I actually think downvotes function to encourage good discussion more often than not. That isn’t true in all cases but it seems easier to be discouraged or get the wrong idea about prevailing sentiments if the dumb comments are just sitting there next to the good comments with no indication that a majority of other users also think they’re dumb.