Latex caulk is crap. No offense, just my experience. I use polyurethane on lots of stuff cause it has quality adhering, non shrinking, bad *** properties.
Most that old sealer stuff just comes off in clumps. A putty knife will usually get the hard dried stuff off quickly. Wire wheel like your doing for the tuff stuff.
So, at lowes and probably other stores is pure polyurethane caulk. Clean your seam with thinner or mineral spirits really super clean and apply nice bead of polyurethane. It will seal and bond the panel.
3m makes great polyurethane caulk for about 9.90 a tube. You will probably need 2 tubes to do the inside. It dries slowly, so give it 2 days before messing around in there after caulking it.
Also loctite makes some great polyurethane (like 6 bucks a tube). I got the white stuff last time but cant remember which product. I just remember it was white pure polyurethane caulk. It dries slowly.. It is high density, excellent adhesion, slightly flexible also.