What would you do?

Sounds to me like you are handling things about as well as anyone could expect given the situation.

Here's my suggestion. Heap fiery coals upon his head. Of coarse this is a metaphor, but the point is, it can be very affective to do the opposite of what you're inclined to do. Instead of responding to his hateful words with similar remarks of your own, show him kindness. Give him respect, even though he has not earned it. This will have a profound effect and eventually he will likely change his attitude. This may difficult to do, and it won't happen overnight, but worth the effort. I've done it myself.

^^^This^^^ He's a sad old man, he's at the end of his life.