Worth the headache?

I can't tolerate the pingy sound of headers.
If you are going to "motor about" more than race, stick with manifolds.
You can make a 318 fast without headers.
Some 340 exhaust manifolds and dual exhaust will be an improvement that still lets you use stock exhaust pipes, run quiet, and go faster.