There is no measurement for a particular case, pinion depth is the measurement of gear mesh between the ring and pinion in relationship to the case. Did you find any markings on the gears? All factory gears should have the pinion depth scribed onto the base of the pinion gear. That is what you're aiming for. Again, gears are matched during the manufacturing process to ensure smooth operation.
Clearly the one shim was not correct. You need to get a shim pack and as stated above, you need to measure pinion depth. However you do it, it will not be correct until you get that measurement right.
Also, did you check you backlash? Spec should be around .008 in. or so.
Bearings pressed on correctly/all the way? Did you use yellow checking paste to see your mesh pattern?
This is not really a procedure where you can just kinda wing it.;)