Mushy brakes 4 life...

The drums drag a touch by hand. If I set the emergency brake, I have a way better pedal. Obviously cant drive it like that. So the drums are adjusted out too far.... but when I set that brake I get a better pedal.

Did you, before assembly put the shoe in the drum to see if it was hitting on the entire surface.

It 'sounds' like the drum may be out of round a bit and the edges of the shoes are hitting and the center is not. Hence the better feel when you have the E brake set.