Do I have reason to be concerned???

In the old service manuals they show grinding them down almost to the axial and then taking a chisel and popping them with a hammer to crack them then they slide right off.

Yup. I'm lazy. I've always just pressed them off. They come right off.