Should I change my cam?!

I think you are really going to like the combination. If your driving a lot of highway miles I'd probably keep the 3.23 gears in it, but if your running more around town and back road miles you might be happier with 3.55 gears. But I wouldn't change a thing without running it first, other than advancing the installed ICL as others have recommended. There are a whole lot worse ways of doing it than your build.