Am I just rusty or is this a problem?

Very typical of wet drum shoes. If a front drum, likely to yank steering wheel to that side. Brake fluid doesn't disappear in a closed system, you have a leak!