It seems to refer to all westerners and it seems to imply "outsider", as in "you don't really belong". A statement of great ignorance, that has become so commonplace.
No worries. Samui is famous nationwide for being one of the worst offended, when it comes to rude and jaded officials. It is normal. So is a delay of a week or two.