318 idle rough after new cam break in and 2 short test drives???

I have a 72 Dart with 318 that idle rough after new cam break in and 2 short test drives, first test drive I went easy on it, but the next test drive I did get on it some kicking in the passing gear up to 4000 rpm, It run fine the who time, even at the stop sign before heading home, got in the drive way, stop to back in and the idle all of sudden dropped down to 500rpm, but revs up to 2000rpm fine, I started it today after setting a day, done the same, idle is at 500rpm about will not run, but get on the gas gos up to 2500rpm it seem normal, but let off gas it dies or about dies! Oil psi cold stat was around 60psi and douse go up with rpm, When I took it for a drive with getting on it some wall driving the temp was 209 and oil was around 20psi!.. Before last test drive it was idling 900 to 1000 rpm & started real good, I do still have the break in oil in it, New Comp Cam (Edelbrock Cam kit) with Edelbrock timing chain, New Cam Bearings, rods and valve seals was put in!.. So I was wondering did I wipe out my cam all ready?.. Could it be a clogged fuel filter, because it has a little stuff in it now, probably from setting a year?.. Could it be a intake leak?.. I'm going to look at it more when I get time!