Driveshaft sanding?

Since I have the driveshaft out to replace the U-Joints I would like to get rid of the surface rust and general road gunk. Any damage to be done by lightly hand sanding and a protective paint coat? I've heard everything from "treat them like a fragile egg" to "hell, just toss em around"......