Engine speeds up when brake is pushed

I am not sure what is going on so thought i would ask all you smart guys on FABO. With power brakes, I thought when you push the brake pedal with the engine running, the engine will drop 1 or 2 hundred RPM's. But when I push the brakes the engine speeds up. Any ideas why??
I should add (just in case it's related) that I am having some issues with the brakes having not enough vacume to the brake booster due to the cam that was installed.