Any Pontiac guys on here? Need insight.

Did you check the Harmonic Balancer?
I'm going back a lot of years. More then 50 to be exact.
I had a similar noise in my 65 Gto.
I would think if it was the balancer that, when we were under it with it running it would have been noticeable but, when under it the noise is much quieter