Cylinder Heads 101

I realize that unskilled hogging out passages can result in worse performance, not to mention the possibility of digging into water passages, etc. The info on shrouding valves using oversize valves on relatively smaller bores was new, but easy to grasp. What I'm wondering is how much difference would a gentle smoothing/polishing of the ports, and matching the ports in the head and manifold or header to the gasket make? That is to ask is such gentle work worth the time and effort for a street motor just looking for a bit of improvement, but not worried about giant numbers???