Pinion yoke cap loose?

If you are referring to the "shield" on the yoke, then you are right- they shouldn't be loose. I have seen these come loose as you describe for reasons unknown. These are pressed onto the yoke. The yoke will need to be removed to be fixed or replaced.