Try thinning your pattern compound. Try just a drop or two of oil mixed with the paint will thin it out and give a better reading. Also, are you prying the ring gear/spool away from the pinion while checking the pattern? Those things may help your readings. I always tried to avoid sharp lines on the witness marks. I haven't done a set in a couple decades, so my memory is a little faint.
