Noisy rocker arm?

I swapped the rocker assemblies from side to side. At first it was distinctly louder on the passenger side. Now it’s just barely louder on the driver side. I’m thinking about pulling an assembly off of my perfectly good running and driving Duster to see if it goes away then. I just hate to open that can of worms.