Every day, I see absolutely moronic comments getting upvoted while perfectly reasonable takes are downvoted. This would be a great opportunity to curate your feed by blocking these users en masse. Active curation like this is the only way to make social media even half-tolerable.
Whether you use it to filter out toxic users or to build an echo chamber, I think everyone should be free to do so. No one should be forced to share space with people they feel bring no value to the discussion - or, worse, make it more toxic.
A simple way is to make downvotes “cost” more clicks. For example:
It isn’t too much of a deal if you downvote people sparingly, but if you’re consistently downvoting others it would get annoying.
Additionally, PieFed has a feature in line with your idea: up/downvoting people gives you “attitude”, and if your attitude is too low (too many downvotes in comparison with upvotes), a warning mark appears near your username. Mods can also use this as a piece of info to decide how to handle you, as users who are consistently downvoting others are typically combative.