People get to wrapped upin the port soize mismatch. A little on low performance engine is not a huge deal. Now go make 450+ hp and you'll stasrt to find short comings.
You're exactly right about that. On an untouched 340/360 head the factory Int's don't hinder intake tract flow, but as soon as you start whipping some air through a well designed Int port, the game drastically changes. While testing a head w/manifold on the bench, some manifolds may actually increase Int tract flow in the low lifts, then become neutral (as the valve is lifted), then the manifold starts hurting air flow. As the cyl head flows better, that dynamic happens sooner.
