Worth the headache?

Usually the first modification I make on a mainly stock car is a set of headers. No better way to increase horsepower and fuel economy for the money!
Headers today are better than the ones that they sold back in the day. They fit better and seal better and if you spend a little more they may even perform better. I'd do it without a doubt.
Without a good free flowing exhaust system you'll never get the benefit of the manifold and carb upgrades. The stock manifolds are just too restrictive. Years ago I had nearly the same combination that you are describing in my 67 Barracuda and loved it until I wanted to go faster. Later we did a similar combo in a friend's Duster and another in my brother's Dart. Definitely go with the headers. Like I said I'd probably do it 1st.