It is a time proven fact that it hastens seal and converter hub wear, and just because you never had a problem does not mean it's BS.
Besides the dust thing, have you ever seen a split bellhousing because a rock bounced up in there?
Or that can't happen either because it never happened to you? :D