Trick to Removing Parking Brake Handle

I have a rather simple question. I am looking to replace my worn out parking brake handle on my 69 Dart. The replacement one clearly has a pin (like a tensioner) that passes through the handle to hold it on the brake rod. It seems apparent that the original does as well. My question is how the hell do I push out the factory pin that seems to be frozen in the existing handle? Is there a trick to this, or just a really big hammer (and maybe a specific tool)? Any advice would be appreciated.