Manual says to use drill bits as gauge but numbers?

You'll notice that most of the number/letter sizes are perfect to use as drills for various sizes of taps. If you work in the machining field these things become etched into your memory.

Most of this "crap" can be blamed on the British. For example the number drill sizes are based on the British "Stubs Steel Wire Gauge" from the 1800s.