Steel or Aluminum?

Yes you can.

The down side is that you lose the inertia provided by the weight of a heavier flywheel.

For example it may want to stall at idle (or just off idle with low throttle) but this really depends on how much weight you cut.