Trying to change turn signal switch

So today I tried to change my turn signal switch. The little plastic piece that holds it down on a left hand turn was broken.

Everything was going well until it was time for me to remove the "bearing housing". How do I remove that thing? I'm trying to get it outta the way so i can install the new switch.

To give a run down I :

Removed the column bracket
Wheel/horn
Old switch/wires


Does anyone have a good link or something with instructions for a noob?