What are the 4.04 pistons for? That diameter would be a stock bore 340 piston or a 0.040" overbore 360 piston but there is also the piston height difference. If they are 340 pistons and you hang them on a 360 crank they will stick out of the bore more than you can make up with a head gasket. If they are 360 pistons and hang them on a 340 crank they will be way down in the hole resulting in very low compression.
As far as which way to go! What ever motor you can get the cheapest is the way to go. Most likely the 360 will be cheaper but there is a deal from time to time on 340's. I just picked up a very low mileage rebuilt 340 long block for zip. Will likely use it for a stroker motor.
Another option is to pick up a late model 360 magnum from a pickup. They have better heads than the J heads you have and are a roller cam motor.