Is this a bad idea?

The question is, would it be a mistake to swap to an OD 4 speed
No!
Maybe 2000 rpm will lug the engine on a hill at 65 mph.
If it does, you got three other gears to chose from..... lol.