Need help gaining access to ignition switch

I need a new ignition switch, and while I'm at it, I have an ignition lock cylinder that matches my for keys I am going to put in. I am having trouble getting the middle section off the column. I thought there was a bearing around the shaft that would slide off with a little persuasion, but the whole shaft pulls up an inch or more. I can see the prize behind, but this piece has to come off. Is there a trick or does the shaft come out with it?1497750090132802237507.jpg