Should I swap gears?

All depends on what your idea of fun is. When I installed my 408 I was blown away at how much torque it has. A buddy of mine has a strong running 440 magnum Challenger with 3.55 gears which used to leave my car when I had a 360 in mine. Fast forward to my car with the 408 installed with the same 3.55 gears. First problem was traction. A set of drag radials helped a lot but I still have to take it easy out of the hole. Even coming out of the gate at 1/2 pedal I can easily walk right away from Bill's Challenger now. With stock heads your going to probably make peak power in the low to mid 5k range so 4.56's would be nothing but tire incinerator's and top speed will probably be limited to about 100 mph.

Also keep in mind that the piston speed of a 4" stroke SB is considerably higher than that of a 340 or 360 so 4.56's with a 4" arm is akin to about 5.57's with a 340. In other words it's a lot of unneeded wear and tear on the engine