So what oil SHOULD be used with a new engine AND why?

if you ask a thousand people, you'll get a thousand different answers. I don't know about break-in oil, but racing oil is NOT the answer for anyone, but maybe true race engines used in racing only. I suggest going to ramforum.com and reading their oil thread. It's about a 1000 pages. Some of it is just back and forth, some anecdotal evidence, but generally some interesting discussion. I also like the oil rat blog. You can look it up on Google. He has some interesting ideas. If you're like me, you'll accept most of it, but make your own version from your own experiences.
I don't like the idea of non-detergent oils for IC engines. Most name brand oils are all good, just some are better. Zinc is good for flat tappet engines, some have enough as-is, some need a zinc additive. But it's not one of those cases where "a little is good, then more must be better, and too much is just right."
I am a believer in good filtration. I use WIX 51515, or 51515XP filters where possible, but I hear there a few other filters which are good, but to me they seem over-priced in comparison . I also use magnetic drain plugs, and Filtermags. Generally, I use 5K mile or one year oil change intervals, whichever occurs first. But I monitor the color of the oil on the dipstick, whenever I fill up. I also, use an oil filter cutter to examine my filters.