Any drawback to keeping the drum spindles on my 66 Valiant?

You can ream the small ball joint spindle to fit the larger ball joint and fit BBP brakes / brake kits.