Idler Arm Removal - Am I missing something??

Jack up the engine. Lots of jobs need other things removed first. You may also have sagging motor mounts causing the interference.