How would you handle

You have his number? Call him. Make it very clear he best not bring any friends to visit. And under no circumstance he drop by without calling first.
If he does call, use excuses like firearm training or target practice.