any benefit to swapping heads?

I am not knowledgable mechanically so forgive me in advance. I have a 72 demon with a 340 block 69 and 360 heads with a purple cam 282/482. The 360 heads are 1.88 intake 1.62 exhaust.
I have a complete set of X heads with 2.02 intake and 1.60 exhaust...
this is a 4spd street car with a weiand dual plane intake and 2.25 stock exhaust manifolds. 355 rear sure grip.

Will I have any noticeable power/torque swapping the heads?

Nice! It's up to you but I wouldn't worry about 20 horse. It is perfect the way it is.