Idles fine until you tap brakes

I am guessing it has something to do with vacuum? The car is a stock 318 2bbl and it idles fine until you touch the brakes. Then it idles rough until you let off the brakes. We thought it might have been in the power brake booster, change that it still did it. Put the original booster back on and checked carb lines. Still have not found anything. Any suggestions??