When to worry about engine noises?

These fast rate of lift cams are known for making noise
My RB 500 has a high lift cam hydraulic. After break in, it sounds almost like marbles in the engine or a solid cam. Scared the hell out of me. I went so far as to change to different set of lifters. Still the same. Car runs awesome. It's been 8 years now. I picked up a solid cam and lifter set for it to put in this year really only to make it sound normal if that makes any sense.