You are looking at ethics and morality from an individualistic perspective, not a systemic perspective. People opposed to landlords are more concerned with the latter. This is an insignificant edge case needed to construct a situation where the individual and social ethics diverge and has little relevance to policy decisions.