is this wrong...

I think at one time or another EVERYONE has done it. When we were younger exhaust used to rust out. The first thing you did was have duals bent up at the muffler shop. Not everyone could afford headers.

You'll still get a performance increase, just not the maximum amount. Have a H pipe welded in, that will help. Save your money and buy good mufflers, you can always add headers later.

X2 Nailed it