Does the primer color make a difference

Absolutely. Candy apple red is a multi-stage color too, each stage effects the final color. If anything, candy apple red is MORE susceptible to changes because of the primer color than a standard solid color.