How bad is this?

With the way that the quarter panel is buckled by the tire, I would recommend having the frame checked to make sure it hasn't bent... Most likely it bent/buckled the frame on the top of the curve where it goes around the rear axle... If it is bent, a good frame shop can pull it straight...

I got under there yesterday and tried eye-balling it to see if that part of the frame you're talking about was bent and it seemed pretty straight to me. I will definitely have someone with more experience check it out, though.