Somebody swapped most of a '77-'83 "Super Six" 2-barrel setup onto your car, and didn't put in the complete kickdown linkage.
Super Six kickdown assemblies have grown rare and costly. See
this thread for kickdown options.
In the meantime, no need to be afraid of driving it even at normal road speeds; just shift manually up through the gears—don't put it in Drive until you've accelerated through 1 and 2; you'll put unnecessary wear and tear on the transmission. You don't have to downshift through 2 and 1 as you slow down; you can come to a complete stop and then shift to 1 to start out again.