Gear Shift Handle Removal

Ok,
So how do i remove the gear shift knob from the floor shift lever/handle. i see a small allen set screw and i can remove that but the knob just swivels but doesn't come off. any thoughts or ideas?