I hate being called a....

Your overly sensitive on this.
Also, IMO, what you should / could have done is said what you said here. To the track owner I would have said;

“Thanks but I just wanted to take photos for my own self and not be a rep or official photographer for the track and/or the drivers.”

On taking orders on where to be and what to do, hand out trophies, if you don’t want to do that then tell him so straight away and promptly. Be Gracious and thankful to him. And let them know that you still want to take pictures, but, not as an official photographer for the track.