Why do I have such sloppy steering?

The idler arm is always an issue with bump steer.

1st get a new one.

2nd
Look at the idler arm while someone turns the wheels back and forth, also check see if the idler is at the same angle up/down as the pitman.
If the idler is not angled identical to the pitman and most aren't, you elongate the idler bolt holes, put thick steel washers over the holes, insert the bolt and snugged so that the idler can be positioned at the right angle marked and or washers welded.
After that a toe set at the least needs to be done.

Well thats what I would do at least.jmo