Changing Cam Bearings

I took the cam out of my 5.9 mag the other day, and all the bearings look great except the next-to-last bearing in the rear. It literally looks like some aluminum-loving termites made some trails on it. My question: can you swap out cam bearings with the rotating assembly in place? I have a driver/installer, and I have done a few sets on bare blocks. I have it torn down to the shortblock, I really dont want to tear it all the way down.