Grinding sound when starting, only once

Had the Dart out yesterday. 3rd outing this year. Drove about 15 miles total and 2nd stop was to eat. Got back in the car after Arby's sub (mmmm) and hit the key and heard a sound like chunks of metal the grinding of body parts and release the key as fast as I could. Figured I would try it again being as I was stranded anyway and it started just fine. Went straight home and it has started three more times just fine after taking a external look. I have the small starter motor with the long rods and they are tight. Nothing looks out of place. the tranny was rebuilt by the prior owner a couple thousand miles ago. Do I have anything to worry about?