Most likely the diaphragm in the booster is shot.
Make sure you have sufficient vacuum going to the booster and that's about all you can do without replacing it.
The good clue you gave was that it seems to have boost at first.
This tells me the rubber diaphragm is split and when it starts to move that split opens loosing the vacuum.
Engine idle probably changes when you press the brakes also.