It will do that to some degree even when things are right. You definitely want possitive caster, I say 5-6+ others say 4+ max. If you think it drives good now just think after its aligned RIGHT how it could drive.
I have the RMS upper arms in the car with the rod ends. I have the upper control arms installed putting the ball joint as far back as possilbe. im sure i have plenty of caster. i will jack it up and look really close at it all and see what i can find.