oil in the antifreeze?

Based on what you say, I think just like you do.
It smell like trans fluid?
That's a good indication.
I don't know which has the higher pressure, but I guess it would be the transmission pump most of the time.
Be careful when you leak check the radiator.
I forgot to turn the pressure regulator back down when I was checking a heater core and blew it up.
Duh.