End-to-end encryption have been designed so that a "middleman" such as Signal can't read your conversation. Signal goes even further by encrypting metadata protecting other information such as who you're talking too and at what time (some technical and targeted attack could however determined these).

In asymetrical cryptography we tend to assume that what we call middleman is a third-party placed between the two peers during the public key exchanges (such as handshake). Signal is indeed a middleman on the infrastructure level but the software has been designed to protect you from middlemen having access to the raw, unencrypted data.

That say if you don't verify your peer's public key it's not impossible that someone has done a man-in-the-middle attack and that you're sending message to him and he's rerouting them to your peer, etc... However this is unrealistic for the average person.

So even if it's not a p2p infrastructure but some centralized servers we can assume that there is no middleman thanks to e2ee.