Mainly because people who are anti-immigrant moan about "unbearable" rates of immigration at immigration rates far below 1% of the existing population and have prejudices like the ones in your comment about immigrants being criminals or replacing the local culture or language even at those comparatively low numbers.