Planning to restore factory AC one day

You would need different (shorter) belts, and you would need to remove the A/C compressor (you can't just bypass it)

But in short, yes
I've got a 69 Chevelle A/C car and have been running without a compressor for years. It has a three groove pulley on the crank, it shouldn't matter.