Things that are confusing answers

Why are women called "The fairer sex" when my wife insists my sex was only "fair"?