Do I really need a stroker?

I know this thread has been posted a million times but I just really want to here some opinions.

Current engine is .30 over 360
Edelbrock heads
Performer rpm intake
Flat top pistons
Mp .484 cam

I am at a point in my life where I can afford to build a fordged 408. The problem is that I am extreamly hesitant to spend any of my money and this swap would mean spending well over 5 grand by the time I buy a new rear end, new torque converter, and the engine itself.

The car is mostly a street car that I have been taking to the track about 5 times a year. I like the idea of having 500 hp but I also like the idea of spending 2k instead of 5k. If I rebuild it at stock stroke then I am changing the cam and porting the heads and adding roller rockers.

I just can not make up my mind and it is driving me nuts.