The reason asexuality isn't misunderstood that well, I'm guessing, is because there are different forms of it.
Many asexuals don't get turned on, and these are called non-libidoist asexuals. Many get turned on by people but are asexual because they don't get turned on by the act of co-pleasure. These are called libidoist asexuals. Some not only don't get turned on but are averse to what others would consider physical fulfillment.
Unlike other orientations, where it's typically a fated circumstance, because asexuality pertains to a lack of what it refers to, it's equally possible to be born with asexuality as well as acquire it later. Someone not born with it could be rendered asexual, for example, through a virus... correct me if I'm wrong but I read somewhere that long covid had a symptom or two like that.
In any category, you can also be asexual and still like companionship (romantic asexuals), or you might not (aromantic asexuals) since companionship is fulfilling another part of us.
Suppose I was facing some kind of Journey to the West type of trial and someone wanted to tempt me, so they bring in some guys who they think are a surefire way to get me to become tempted. It's not a matter of self-discipline, it simply wouldn't work as they envisioned. I would walk right through them unaffected. That part of me is to physical temptation what a colorblind person is to color.
That's why sexual orientation is often told of as a discovery process. You will hear people mention when they "discovered" when they were this or that. In a society without implied bias, there wouldn't be any such curveballs though.
A friend of mine who is also in the asexual crowd came from an enormous family and often observed their behaviors and habits and wondered about them, and conversations came up about it. He, being five years older than me, was surprised at the time to have a conversation with a relative of his who adhered to all the incel stuff that physical expression in a relationship wasn't just a form of social norm rebellion taken up by mischievous coming-of-age individuals and that there was a drive towards people acting out physical expression. The relative, as well as many others then and now, were equally surprised this friend of mine had no outright appetite or even desire for physical expression, instead, like me, preferring company alone.
Legend has it this relative, who made my friend realize he (said friend) was in the asexual crowd, was trying to woo my friend's would-be lover (who is my other best friend), but in causing my friend to realize he was who he was, led him to revealing to me that I was the same based on me exhibiting the same differences in behavior and habits, and I in turn did the same for the would-be lover, since she happened to be one too (something in the water I guess?), with the would-be lover strengthening her bond with the first friend as a result, since those in the asexual category often find themselves unable to maintain relationships with individuals who thrive on physical expression. This, in turn, was said to erupt the ire of the relative, who, in not understanding what asexuality could possibly entail, put himself at a disadvantage in regards to the friends.
We do vary, yes. But that doesn't mean someone's asexual status can't still be determined, even at a young age. It's not as if people at young ages haven't grasped at least a little of their "interests" and how they work... to say someone's not old enough to know a little of how their interests work requires one forgets about this. There is a difference between being interested in something out of an inclination and because it's the social expectation, which plays into how people of less represented orientations "discover" who they are. The asexual "nicene creed" or "binding omen" is when you see someone who, when put in circumstances where that physical expression is impossible, such as an abbey or in a prison, doesn't care whatsoever.
It's only difficult to understand why someone doesn't have a certain inclination because of culturally-pushed idea... nobody is saying to those in the depression crowd "I can't understand why you don't feel happiness" and nobody is saying to people outside the periodic cramp crowd "I can't understand why you don't feel cramps", but they all are saying to the asexual crowd "I can't understand why you don't have this inclination", which is an irony that's added onto when those same people say "well it varies, how do you know you're one" as if they didn't just allude to circumstances when the inclination isn't on the table. Even when someone is gay nobody questions it, but for some reason asexuality confuses people? In all due respect, their confusion confuses me.