Yeah the short answer is almost all the sheetmetal on these cars is fairly vital. Some is obviously more important than others, but being a unibody if you start taking out too much sheetmetal things get flexy in a hurry. Heck, even from the factory they were pretty flexy. It wasn't as obvious back then because of the lousy traction the bias ply tires had. But even with your basic all season radials that most guys run now you can put more flex into them than the factory designed for. Hence lots of discussions about how to make the chassis stiffer.
I ran my Challenger around for 70k miles with holes in the floors (and other places I didn't know about). The car didn't fall apart, but, the chassis flex was really noticeable. I probably should have pulled it off the road long before I did. Did I die? No. Was it a good idea? Nope, not that either.
