I saw in an earlier post that some of your family are upset that you are spending so much time with your brother... If your brother doesn't have much time left, then make the most of it while you can...
You keep by his side, the rest of the family will be there when he's gone and you can make up for the 'lost time' with them later...
