Is this something i should be concerned with??

Its a wasp of some sort. The exterminators use a special vacuum to suck them out. They usually have something in the vacuum that kills them. Wasps go somewhat dormant and slow at night. Get them when its cool and there is much less chance of getting stung.
This is a fun tool to use on bees. I use mine on hornets when they come in the garage.
Propane Torch