How the F#@% do I remove it?!

Hi All- trying to remove the broken parking brake handle from my '64 Dart. It's just a handle and pin holding it in place. But I can't get the friggin' thing out for the life of me. I thought it was as simple as pushing the old pin out, remove the handle, put the new handle on and tap the new pin in place.

Is there something I'm missing? Or did I just not swear at the car enough for it to come out?