Whew that is a messy situation. Sorry to hear it. Well, I gues you won't mow the lawn there anymore.
Does this so-called dad do any work to maintain the property? Or are you doing it all? If you are doing it all.... then speak to your lawyer about something call adverse possession. In VA basically after 15 years of paying taxes, maintaining the property, doing so without permission (that is the sticky point here), and doing it in a manner where the world can see it, then you can make claim for 'quiet title' to the property.. In other words, you can claim the property.
Sounds like you have met most of the requirements for this: paying taxes, maintenance, and doing so openly for the world to see (legally termed 'notoriously'). The thing to focus on is if this was done 'without permission'. If there is no written agreement on this and no witnesses to any permission being given, that might put you in a good position. But, if he has kicked you off the property now, then that may put the last nail in this idea. Understanding this permission matter is where an experienced attorney's advice is essential.
But I can very well see also see chucking the whole deal and moving on. Sounds like you have had confrontation enough for a lifetime in this area.