904 shift problem

OK, so I have had some time to put some miles on my Dart with my funky 904 I built. I did the gear swap so that my low gear is 2.74:1 and second gear is 1.45:1 with drive still being 1:1. ATI Trans Scat shift kitted. Used the govenor out of the tranny I got the gears from, a late 70's 904 (was supposedly supposed to be able to get them from a mid 80's A999 but just wasnt where I found mine.)
So anyway, when driving on the street if you run it at part throttle ie just easy on it, it shifts very very harshly in the 1-2 shift, kinda a thunk. Same with 2-3, harsh and thunks almost sounds like a u-joint clucking when it shifts. BUT if I get on the throttle, like half throttle or more, each shift is FIRM AND SOLID, no harsh clunks or anything, just shifts the way I expect them to be. Fluid level is as it should be. I always let her warm up a good few minutes before even thinking of putting it in gear. Oh also when putting it in reverse it is rather harsh also. Any body got any ideas on what may be going on or suggestions about where to start looking?
When I originaly got it together I had troubles with it and had to take the damn valve body in and out about 4 times just to make certain that we hadn't missed something that would explain a late shift problem I had. Finaly narrowed it to the govenor. In trying to find that problem at some point in one of the times of having the valve body out we took the restrictor plug with the small hole in it, that came in the ATI kit, out of the hole that the kit had said to put it in. We had thought that it might be the cause of the late shift. That hasn't been put back in yet, is it possible that this may be the reason for the light throtle harsh shifts?
I know it's hard to make an assesment of a problem over a bbs, but any help would be greatly apreciated. Thanks.
Oh and one more thing, when I run it up to drive it shifts into drive at about 33-35 (unsure as the speedo aint quite right) but if I slow down about 5 mph it drops back down into second gear. I can't get it to run along on the secondary streets in drive, and the local fuzz dont like you ripping along in a 25 at 35 mph, and if I run it in second it's just to much rpm at 25 to 30 for daily driving, hard on gas and the exhaust tends to be a little noisy.