Holley Street Avenger 570 Idling Problems

Yes, the cam is very mild. When the temperatures were warm, I could idle it down around 600 rpm and drop it into gear without any issues.Only issues at that time were the ported vacuum port for the vacuum advance had full vacuum, so I was just running it with mechanical advance.
I'm trying to idle it around 700-750 or so in gear, something the engine would be happy with without having problems holding it with the brakes.It does have power drum brakes.
I had returned the carb to original specs, since this engine is slightly different. The cam was degreed in when installed. I have not run a fuel pressure test, or a compression test or leakdown test yet. The engine has 0 miles on the road since being installed. Thought everything was good to go until the weather changed.