If you have a 4 prong ballast resistor it could be a bad connection there as well. If you have corrosion or a bad connection it will act like you describe.
I don’t think the ballast resistor has anything to do with the starter engaging. The ballast resistor can cause a no start issue, but not a no crank issue.
Now, I am reading into it as the OP isn’t spelling out which it is, but the fact that he can start it at the starter relay implies (to me) that the problem is a no crank issue.
