How much drag should there be rotating my rear wheel in neutral. My brakes are not dragging

Yes I do have sure grip and 3.23 Gears but the force to rotate seems like a lot two hands
It will be. Think about what you're doing. You're turning BOTH tires, drums, axles ring gear, pinion AND drive shaft and you're turning it BACKWARDS from the normal direction of power. Let's say you have a 3.55:1 gear. Turning it from the tires, now you have a 1:3.55 gear. Understand? MUCH less leverage than powering from the drive shaft.