Removing Early A inside door handles?

How do you remove the inside door handles from an early A? Taking out the allen headed screw and pulling alone doesn't seem to work very well....what am I missing?

I thought I had seen a post about this but I couldn't find it.