A lot of people here mentioned that passwords are hashed, but unless I missed it no one pointed out the following:
The admin of your instance controls your login form and they can pull your password when you log in. So, as others mentioned: always use unique passwords, never ever reuse them.
In general a server admin can do anything they want on their own instance.
Federation wise I'd say if your home instance is the bad actor you are screwed, if it's another instance then their capabilities for mischief hare probably (hopefully?) more limited. And any such action would likely cause a swift defederation of the malicious instance