To be clear, the issue with IPv4 being preferred over ULA is only an issue with calls to things like getaddrinfo which can return multiple results for a single host/domain name query.
The solution to this is easy — either don’t put your IPv4 addresses for hosts in your DNS, or give them a different name. I.e.: maybe you’re using hostname.domainname.xyz for your IPv6 addresses, and hostname.ipv4.domainname.xyz for your IPv4 addresses. Lookups against hostname.domainname.xyz will return ULAs in this case, as there is no A record for the same domain name.
People (many who should know better) act like machine A can somehow magically determine that machine B has an IPv4 address and just start using it instead of ULAs — but the precedence rules only apply to a DNS query the returns both an IPv4 and a ULA. Design your DNS so that this isn’t the case, and you won’t have to worry about it.