Caster thoughts?

I assume they are offset to the outside. You didn't say, so from here, I may be 180* off base. Offset wheels to the outside of the car effectively increase SAI. It really has zero effect on caster, although it can seem like the same thing. Since you're moving the centerline of the wheels inside and the rim and tire to the outside, you're increasing the amount of leverage it takes to steer the car, much the same as too much caster, but it's not quite the same. You've moved the center of the axle closer to the ground which increases the SAI arc travel, requiring more effort to steer. Reducing caster may help that, but it will probably give a bad road feel. You may not be 100% happy with it until you switch to power steering. Since SAI I a built in angle and non adjustable, there's nothing you can do except change wheels.

I would think the short tires would aggravate the problem, as they move the center of the wheel even closer to the ground effectively increasing SAI even further. I could be wrong.

All this is of course if the offset you mention moved the rim and tire to the outside of the car. Flush it all down the toilet if you went the other way.