The comparison with IRC is not very meaningful: moving from one server to another is much easier because IRC users don’t lose followers, bookmarks, posts, etc.

The point is that IRC is normally used in a way that leaves more to the client. ActivityPub services usually expect that users put much more trust in the instances. It might be worth thinking about that.

How would a “built-in protection against centralization” even work?

IMHO, you can only provide tools. You can’t prevent people from being stupid and not using them. That’s also why by now, e.g. the EU tries to solve such problems through regulation.