Dodge Diesel/Cummins Mechanic Advice Needed

The fuel shut off solenoid is stuck There is a rod under the pump pull outto start push to stop. You can operate this manually to start it . but it will shut off on its own with the key.

Sorry I didn't include it before as I forgot to mention, I'm getting fuel to the injector pump from the lift pump. I'm also getting a small amount of fuel out of the lines at 1, 3, and 4 injectors. I"m getting the same amount of fuel that I was before I changed out the lift and injector pumps. Do you think the soleniod is still possibly stuck even though I'm getting a small amount of fuel to the injectors?