Man neighbors suck sometimes. Opinions?

Put up the wall between you and your imposing neighbors and reinforce it at every opportunity. People like them only will encroach further at every chance they get.

One warning and then go nuclear. If they don't realize you are serious about the limits out of the gate, you will never push them back without even more hassle than having done it in the first place.