Of course they do, think of this as a Sears parking lot which you own. If you do a big dangerous smoky burnout on the Sears lot, and "they" don't like it, they will file a complaint with LEO, whoever that might be.
You can NOT deny him access if this is a legal easement/ right of way but you CAN require people to behave ON YOUR PROPERTY