Hardwood Floor Install Help

I did pull the baseboard. The problem is that I have a gap in the center because during installation of the rows, I somehow got the rows tighter and now where I transition into the room there is a gap. I just sketched a quick diagram in paint to illustrate what I did.

The room on the left is where I started. I couldn't verify the squareness of the middle room, so I measured from the first board that transitioned into the middle room, laid a 1x4" as a starter to position the row far enough from the wall so my floor nailer had room and laid the floor. I need to remove the gap (red line in photo), but would rather not rip up the wood I have laid. I was just hoping that there was a way to shift the wood up 3/16" of an inch. Otherwise, I'm going to be ripping up about 15 rows that are laid with 2" L-cleats.