Thanks guys, just had a bunch of parts replaced and when I went to have it aligned the guy told me it had to be replaced before he could align it without telling me the name of the part.
The idler arm has two wearable parts to it. The pitman arm has 1 wearable part. So if the idler is worn at the pivot and not the ball socket would you expect the ball socket at the pitman to be any looser? And the ball sockets should have grease fittings which should extend their life.