Conversely I don't know why anyone needs a passenger side mirror to drive?!? If you truly know how to drive you really only need the rearview mirror inside your car...
Technically, I think you're right but side mirrors really are useful for maintaining awareness of all the traffic around you on a freeway without craning your neck around.
IIRC, it is (or was) a driver's test question that we're supposed to check all mirrors on a regular/very frequent basis. I can't remember the interval now but it was a lot...every 10-12 seconds?