A TPM is a very slow and dumb chip: It can hash data somebody sends to it and it can encrypt and decrypt data slowly. That's basically it. There is no privacy concern there that I can see. That chip can not read or write memory nor talk to the network.
Together with early boot code in the firmware/bootloader/initrd and later user space that chip can do quite a few cool things.
That code will use the TPM to measure data (create a hash) it loads before transfering control over and then unlock secrets only if the measurements match expected values. There is no way to extract that key on any system with different measurements (like a different computer, or even a different OS on the same computer). I find that pretty interesting and would love to use that, but most distributions do not offer that functionality yet :-(
Using the TPM to unlock the disks is just as secure as leaving the booted computer somewhere. If you trust the machine to not let random people log in, then TPM-based unlocking is fine. If you do not: Stay away.
Extracting the keys locked to an TPM is supposed to be impossible, so you do not need to worry about somebody stealing your keys. That alone makes TPMs very interesting... your own little FIDO tocken build right intomyour machine:-)