what voltage should I have at the coil

it should ideally be around 7 to 9 volts depending on the ballast resistor.

You have to be careful making these statements.

If the engine is NOT running (or not charging)

or the coil is a "heavy" HP coil IE draws a lot of current

or if it's cold weather, and or the battery is somewhat low on charge.

coil voltage can be quite low. And, it depends on RPM, too Some run as high as 11--12 with the engine running and system voltage at 14, depending on RPM

But with the key on / engine off, I've seen coil voltage as low as 3-4 volts

But this is a poor place to start troubleshooting. Coil + voltage with key on really tells you very little about whether "it will work" It DOES tell you "it's getting power", that it's properly drawing current through the coil, and that of course the ballast is good. But the actual specific voltage will vary