THIS, stop pussyfooting with this nut...
Look him in the eye and step forward not backwards and lay down the law.. And if he shows up with anyone tell him loudly (to make them uncomfortable) you told not to bring strangers...
Now leave and don't come back without calling first..
If you don't.. enjoy his company and the people he feels like bringing over...