Am I just rusty or is this a problem?

Sounds normal. You don't mention disc or drums but in either case, it sounds normal. You MIGHT inspect the brakes to make sure you don't have a leaking axle seal or wheel cylinder as either will cause brakes to lock prematurely.