Every time people lament changes to the lore that amount to "not every member of species X is irredeemably evil" and claim the game is removing villains from it, I think how villains of so-caleld evil species fall into two cathegories: a) bland and boring and b)have something else, unrelated to their species going on for them, that makes them interesting.
I think a huge problem with this is trying to frame everything through D&D as well as our perspective. Fuck modern D&D and its desire to control the entire dialogue. Wizards of the Coast aside, there’s also a fantasy component here. I personally dislike requiring all races to act exactly like humans with human motives. From a specific perspective, we view the wanton murder and sacrifice of wood elves by the drow as a terribly evil thing. From the drow perspective, why can’t the opposite be true? I’m not talking about Salvatore’s one-sided writing that makes it clear the whole thing is a massive con. D&D is very biased toward human motive and perspective. Why can’t both be true? Drow are evil to us and we are evil to them? That’s a much more interesting story and completely changes the narrative around someone like Drizzt.
This is a really nuanced take on speculative fiction in general. I also strongly feel that, the way WotC writes things, removing racial alignment is very important. There is no nuance in their universe. Even when we see other races, we always evaluate their action through a human lens rather than being presented a cogent paradigm contrary to ours.
DnD good and evil are distinct from common usage of the terms; they are cosmic forces, objective truths. Each action reverberates through the higher and lower planes and tilts the scales towards victory for one side or another in their eternal struggle. This lore doesn't leave a ton of room to change the alignment of entire races (and that is by design, structure makes it easier for people to get in to the setting).
But this is just in the established settings, any DM is free to homebrew any setting and justification they like.
Note that I am not trying to defend this as the height of storytelling, it isn't. It is a consequence of how the setting is justified - with deities being active participants, having specific rules for granting and revoking powers, and the physical presence of higher and lower planes embodying perfect conceptions of 'good', 'evil', order, chaos, etc. All of this can be changed, and again any DM is free to change it, but the 'deep lore' of the established settings over the past 40 years is drenched in this stuff.
One way to consider it is simply - the Drow aren't evil because they are Drow. The Drow are evil because their culture promotes actions that align with the literal true definition of evil that is present in the setting. Evil doesn't mean bad, it is just a label aligning with some physical rule of the universe. Just like the positive charge of a proton and negative charge of an electron are labels for physical rules of our own universe. Positive isn't any better than negative, but they are inherently distinct.
What you’re describing is closer to the nuance I’m interested in than WotC’s settings. If you read some of the later Lolth stuff, it’s the exact opposite of that. Evil is bad and the justification for anything always involves this trite movement from evil to good. They’re not presented as counterbalances or equal combatants. Even evil characters seem to always be working under the assumption that good characters are ultimately better.
The 40k universe has a lot of similarities. However, I’d argue its authors are somewhat better at presenting why Chaos is an equally valid choice or why the Orks can do whatever they want. There isn’t a clear choice (some authors are fucking terrible at this and drive WotC-style to the goodness of the Imperium).
The only reason WotC has to remove alignment from races is because WotC has made it very clear there is the thing people should want and there is the thing people should not want. That’s not an even layout of nine alignments. That’s a huge bias and all of their content reflects that.