Is this something i should be concerned with??

If you decide to try and kill them, do it at night when they are all in the nest.
Because they are ground wasps, I would spray the holes with proper stuff(not motor oil :realcrazy:), coat/spray one side of a sheet of plywood/Osb and lay it on ground over holes. Wear protection and run like the fn wind.