1974 318 duster. Can I port/polish these heads and get enough power out of them?

There is never a scenario where back pressure is a good thing. Tubing diameter and length affect more than scavenging. Cam timing will affect which exhaust sizing will affect combustion chamber scavenging much more than sizing. Get that wrong and the engine will be unaffected by whatever exhaust is bolted to it.
I'm kinda in agreement. Every single time without fail when I've gone up in header tube size and or exhaust pipe size, I've always gained more power. With no exception.