Those parts are often sintered. Letting it self clearance is asking for something to shatter.
Grinding on metal can change or alter the magnetic field due to the heat, vibration, or both.
If I were to do anything, I would clearance it with a fine toothed file and go slowly.
It's weird it's only as if it hits in one spot like one long tooth on the outside and one long tooth on the inside they like to touch each other....
I think this is the way I'm going to go like you said just try and file those two that like to touch each other..