I would leave it, I would'nt try to remove the orange peel either, once you cut it with a fine sandpaper, it's never going to be the same and will be hard to buff and you risk burning through.
Just leave it, noting wrong with a little orange peel, lot of cars were sold new with some orange...