To much cam??

Power brakes need vacuum around 10" at idle to work dependably. They are the limiting factor by far. In terms of idle quality - If you want brakes, ignore the idle quality. If you want the raspy idle, replace the brakes with non-power assist or hydroboost. The cheapest/best overall way out IMO is to replace the camshaft and be happy you can stop rather than shake the ground.
But - that being said - if you do the pressure testing it might be that the cam or timing set is screwy and the cam is in retarded regardless of the dots. A very lazy 383 will wake right up with optimized cam timing events - even using the factory 383HP camshaft. I've had to move them as much as 8° (not a Comp...) to get the ICL in the right place according to the piston - not the dots.