Annoy is completely the wrong term. You're getting to control over what is to be built and what's not, and since softwares are compiled and optimised according to my hardware, they are lighter and faster with less attack surface.
That may sound cumbersome for a novice to setup the portage configuration but in return it is really worth the time, and it is usually one time, unless you plan to add or remove features. But once you're satisfied with your configuration, you don't have to look back at it.
I found YouTubers complaining about going through hour long upgrade on the daily bases very misleading. Only a few core packages can take that long, which are upgraded on a quarterly bases.
Wait did you seriously called it a hype? Before switching to Gentoo, I was using Arch, softwares have better support of eachother and if feature isn't working you can always talk with the dev how to resolve it. They might even look into modifying the ebuilds to make them compatible.
FYI, I never came across any breakage and I've been using Gentoo for about an year now.
Go through this https://www.gentoo.org/support/security/