:cussing: Unless you take it off the engine, you're screwed, Glow Worm. I had the same problem on my Jeep, and I realized that unless you want to spend hours with a toothbrush and some rubbing compound, there is no effective way to do it.
The problem with the Edelbrock manifolds are that they are rough to the touch, and anodized/coated. The best way to keep a manifold clean is to polish it and powdercoat it, or just powdercoat it with clear or the color of your choice. Painting it is another option, especially if you can build up enough paint to make everything smooth( er ), like what Eldubb, DVJ and AbodyJoe do to their engines.
Unless someone has a better solution, get ready for temporary intake removal. Dirt/oil/water/other clings to the gritty surface and bakes, which should explain the brown spots.
P.S. What are the specs on that motor, and how much power is it putting out?