When to worry about engine noises?

I recall a U-Tube video of "Nick's Garage" where he couldn't establish over 25 psi at idle in a 318 he was trying to dyno test. He eventually established that the engine had at one time been fitted with AMC style hollow pushrods, which drained off oil pressure to the point where overall engine pressure was too low.
Not sure if this applies in your case, but may be worth considering.

I'm sure that's part of the issue as I'm using factory Magnum roller lifters which oil through the pushrods, along with hollow pushrods to give extra oil to the pushrod cups in my factory stamped LA rocker arms. I'm doing that because the engine has a .550" lift Racer Brown custom grind hyd roller cam and from what I've read here, stock stamped LA rockers are borderline for that. I just want them to last until I have the money for some decent roller rockers. I do know though I can restrict the oil feed to the shafts because they're getting constant full-pressure oil unlike the factory setup which is "timed" through the holes in the cam journals.

Also when I put together this engine I mistakenly cheaped out (in hindsight) by not putting in new cam bearings as the ones already in there didn't look too good. In fact I did nothing with this high-mileage 5.9 Magnum short block aside from install new rod bearings. Still runs like a champ but I knock on wood every time I say that lol.