how bad is this and how much performance am i losing?

I'd say if it was bad, you would have broke it by now.
Granted this with mild cams and usage, but I will just swap retainers and leave the valves. Especially if it's on the car and the cam is mild. I never measured, but I think the overall difference is less than .100" between the rotator valve grooves and the non-rotator grooves. Biggest cam I've done that with is about .480 advertised lift, so with pushrod angles it's less than .460 at the valve.