How much drag should there be rotating my rear wheel in neutral. My brakes are not dragging

It will be. Think about what you're doing. You're turning BOTH tires, drums, axles ring gear, pinion AND drive shaft and you're turning it BACKWARDS from the normal direction of power. Let's say you have a 3.55:1 gear. Turning it from the tires, now you have a 1:3.55 gear. Understand? MUCH less leverage than powering from the drive shaft.
Yep, I just pulled a 3.23 suregrip center section from my car. This is the 3rd or 4th time trying other ratios. They all had quite a bit of resistance turning that way. Turn the driveshaft yoke and a different story. One direction is always a little easier than the other also. It’s normal. If you’re not getting any noise while running it , you should be ok.