Which part does the "wearing"?

Just from observation of worn parts, they both wear. When something goes bad, you can see problems on both parts. The cam surface is hardened, and so always seem to wear more slowly (again, just based on what I have observed on used parts).

You can use new lifters on a used cam because the new lifters have fresh convex surface on the bottom, that will promote the proper rotation when working against the taper across the cam lobe surface, and a performs new break-in process. But if the taper is all worn off of the cam lobe, then the cam is likely too far worn to re-break-in.

Interesting discussion here:
http://www.enginebuildermag.com/2009/04/flat-or-tapered-cams-and-flat-or-crowned-lifters/