Thats what I say.....way easier to ask him to come over for a beer. When he declines say OK then ask him again teh next time you see him.
Killing him with kindness will show him your not a bad guy and if he has any sense of self worth he will accept. You can find out a lot about a person over a beer or two.
He may be lonely and jealous of you and your sons.
Keep turning the other cheek and know you are the bigger man for doing so.