Hate to do it, but...

Anything that invades your airspace is yours to remove and your neighbor has no say in it. Even if the base of the tree is on his land you have the right to remove any part of the tree that goes into or over your property. But since you are good neighbors it is best to just apologize hope everything is forgotten/forgiven

I let it go and explained it off as a miscommunication, since I told him where I would be cutting prior to having done it, even of he didn't remember.

Just looking, really, to save any future frustrations on misconstrued border lines.