Which part does the "wearing"?

Different cams are made from different materials too, so I'd say it's a toss up. Had a friend with a late 70's Vette that was known to have soft cams. Ground a lobe down to a circle so the valve wasn't doing anything. Cams can be cast and billet as well, which I'm sure has some effect on their durability.