How would you handle

Had a similar circumstance. I just said I enjoyed his company, but I really needed a heads up before a visit. And sometimes when he called, I just said I was really busy, and right now would not work. But every once in a while, I'd call him up and say let's go to lunch. Worked out fine.