Not always. A center counterweighted crank is the perfect example.
If the crank is flexing all over the place (and they do) and you can add center counterweights, you can tighten up the main clearances and the car will be quicker because the crank isn't moving...as much.
That's a perfect example of where to remove weight and where not to. Wrist pins are another place to not skip. I has a BBC on the dyno and after about the 5th pull it started losing power. A couple more pulls and I called it off. Upon tear down, you could see the pins flexing and pulling the Pistons out of shape. Since I made the call on the pins, I got to buy a new set of pins and Pistons.
A lot of hype and BS out there about saving weight. All weight savings are not good weight savings.
One more...the last guy I worked for built himself a super lightweight everything for his circle track car. Conduit sized wrist pins, an ultra light crank that had even more weight removed after he got it (I forget how many slugs of Mallory he used to balance it, but after the experiment he kept all the Mallory out of the crank, and for over 3 years I didn't have to order any Mallory...I used what he had left over!!!) super lite Pistons...all the trick junk. When the car hit the track he said it was like he popped a parachute half way down the straights. The engine didn't have enough internal weight to overcome friction and drag. They played with it for awhile, because on the dyno it made big power and looked good.
I forgot to mention he also had an ultra lightweight clutch in there too.
