Well whoop de do.
Wanna know what i was trained in before i joined up with the Air Force.
Toolmaking, in factories back in CT. when manufacturing was still done in the USA.
Using Emery cloth on that spindle surface will have such a slight tolerence on that spindle it won't ever make a difference out there in the real world.
The op isn't building the next space shuttle.
But you did clearly state it will remove tolerances to a professional that might be very light but the somebody that doesn't realize that they might be sanding a little harder or unevenly more than normal and maybe removing more than they expected.