You can never know. If there's ever a doubt, do a factory reset and start from scratch. Keep the phone secure keep the password safe. Use biometrics to reduce your risk surface of people shoulders surfing your code.
The same goes for any computer once it's been compromised you can't trust it at all. If you want to be safe you start from scratch