VIN tampering is fraud. Period. Would you pay 4 or 5 times as much for your car just because? No? What about only twice as much? That's not that bad right? You'd pay double for your car for no reason, that's not a big deal. If you sell a car that's had its numbers altered you've committed a crime and deserve to be in jail.
Personally I think the numbers game is silly. Lots of those high dollar Hemi cars have less original metal than even my Duster/demon clone has. So what are you paying for? The right combination of stamped numbers. But if they played by the rules then it's ok. That's the way it works. And there a plenty of legal ways to repair damaged cars without just swapping VIN's. They involve more work and require more talent, and that's why lazy, untalented people that want big profits do body swaps.
Somebody gets left holding the bag. But I'd rather be the guy left holding the bag and lose that investment than be a dirtbag that misrepresents a car and loses my integrity.