Agreed. The stock servo is prone to cracking when the pressure is cranked up and with a manual VB it's cranked up pretty high. I'd upgrade it.
What stall speed converter are you running Josh? If it's a stock or real low stall the shifts will probably be bone jarring harsh. If you've got at least a 3000 stall it'll absorb a lot of the harshness.