Weird cooling system behaviour

just my guesses.
1. the brown crap is stopleak.
2. the radiator and heater core are probable shot
3. the stopleak has plugged your engine and you will have to flush several times after you get the new heatercore and radiator in
4. All this is going to be expensive.