How far is the belt moving/hopping/whipping back and forth? (It is hard for us to judge from the info given.) If not much, and if it was not moving fore or aft, then I would not do anything. There will ALWAYS be a 'least tension' section of any belt (it depends on where the highest torque loads are located) and any tiny bit of slack will show up there. Putting a fixed tensioner at the wrong spot may cause belt breakage, as it can cause some really high instantaneous belt tensions when the belt hits it.
If you still wanted to do anything, then I would look at that new tensioner first; so many after market parts are made with the incorrect springs or rubber durometer levels, so it could just be too weak. Perhaps look to see if you can move the anchor point of the tension spring? Or get an OEM tensioner.