Left and right are referenced to yourself, or relative. Port and starboard are referenced to the ship, or absolute.
It's the same reason people who work in theater say "stage right" and "stage left" instead of just "right" and "left."
Imagine being on a boat for weeks, in the middle of the ocean, with no land in sight; it makes a lot of sense that you would think of locations in terms of the ship, as that is your only reference point.
Left and right are referenced to yourself, or relative. Port and starboard are referenced to the ship, or absolute.
It's the same reason people who work in theater say "stage right" and "stage left" instead of just "right" and "left."
Imagine being on a boat for weeks, in the middle of the ocean, with no land in sight; it makes a lot of sense that you would think of locations in terms of the ship, as that is your only reference point.