New exhaust system rubber has come out on hangers

On my new exhaust system the hangers at the end of the tailpipe are having the square rubber insert coming out of the square tube, and the metal pipe is rattling around in that tube, and I can’t seem to force it back in. Anyone ever dealt with this. The exhaust systems less than a year old. If I can fix this without welding that would be ideal, because I don’t have access to one
Push it back in then drill a small hole and put a screw through it or you can slot a large washer stick it up against that bushing on the hanger rod and weld it.