Looking for valve train noise help!

Ok, here's the problem. I bought this '71 Duster with a built 340 in it for a reasonable price. It had been raced by previous owner I was told. The 8" turbo action torque convertor fried on me causing me to have a more streetable lower stall convertor put in. The car wouldn't idle worth a crap afterward because of the large cam and smaller convertor and I had to 2 foot it when stopping. I ordered a slightly smaller cam (Comp cam Magnum) and new hydraulic lifters and installed them. The car seems to run fine and still plenty of power but I cannot get the Crane Gold roller rockers to quiet down! I followed thier manufactured guidlines to adjust them, tried to adjust them according to Comp cam specs and even tried 0 lash +.20 they are just noisy now. Plenty of oil being pumped.Any ideas would be appreciated. Is it mandatory to face the oil hole of the lifter upward when installing? New cam has .480 lift, kept same pushrods (upon advise from summit tech)
Thanks