Steel or Aluminum?
Yes you can.
The down side is that you lose the inertia provided by the weight of a heavier flywheel.
For example it may want to stall at idle (or just off idle with low throttle) but this really depends on how much weight you cut.