Removing or Installing Lock Plate Pin

I tried driving it out with a large center punch.

I even used wood wedges so the whole thing would not shake. Still will not move.