If it is a driver car and the compartment looks like crap, I would say yes.
If it is a nice cruise in car and the compartment looks good, I would leave it white.
Either way it is YOUR car and it could be changed back if another owner wanted to put it back so do as you wish.