Need advice for 340 intake swap

The port mismatch is much ado about nothing. Ma Mopar herself build millions of 318 engines right from the factory with big port intake manifolds atop "mismatched" small port heads, so they must notta thought too much of it.
Alright great because many people have told me that it will totally kill performance. But if I’m currently running the 2 barrel intake, a 4 barrel has to be better regardless of port size mismatching right?