Panel Alignment-Is it Fair to "Cheat"?

Looks like my doors are setup just about right and I would like to not have to fool with them again. The passenger fender is one micron from the door in two tiny spots, but will pass everywhere else. Might be a little plastic filler on the edge of the door or fender making it such a tight fit in those areas, might not be.

Since I would like to keep the door where it is, and the fender for some reason feels solid and won't budge forward (even with all the fasteners loose) I am tempted just to sand/grind a small amount in the tightest places and call it a day. Any similar experiences out there someone would like to share with the group? Thank you.

(photo not my vehicle, for demo purposes only)

gaps.jpeg