I mean, that's definitely a possibility but it wouldn't be the first thing I went and looked for.
If the wheels have the wrong offset and are putting the tires really close to the body it could also be that the car just sits a little lower on that side. Ride height can be really critical for tire clearance, and small difference in ride height can absolutely make the difference between the tires rubbing or not. It's really easy to have a car sit lower by a 1/4" on one side, especially with the torsion bar adjustment and old leaf springs. Heck it could be 1/8" low on one side (you'd never see it unless you measured) and the wheels are just barely clearing on the other side.
It could also be really simple, like the wheels are close all the way around but the route that the OP takes with his car is always the same and it rubs on the same corner every time because of the speed, banking, angle of the corner etc. If there's a downhill corner with a little off camber road grading in one direction it could easily be the only corner that causes rubbing, and it would always be the same side. Lots of possibilities.
