That's certainly a consideration. Performance wise, several years ago we ran a test on a 400hp car with our 2.5" X connected to LT headers and then ran the same exact test with the X-pipe removed. The car dyno'd 10 extra hp to the wheels with the X-pipe on and nothing behind it! So in theory, you might add weight but you might still pickup a little extra in the quarter mile.
That being said, with X-pipes, there comes a point with really big motors where they may not increase your hp. We never tested a 3.5" X versus open headers. Some guys swear they work and some guys buy them strictly because of reduced decibels.