I never understood it, even as a kid, since we subscribed to National Geographic, and in my family it was accepted that the earth was ancient. It was usually some older guy who'd pontificate awkwardly in F&T meeting. I have a friend from high school who still believes in a young earth, which is especially strange since he wasn't like that when we were teenagers, but has picked up some strange beliefs since then. It's a weird conversation when he brings it up.

Your point is a fair one, though. It's clearly a minority opinion these days. Maybe a better example would be how LDS and ex-Mormons interpret historical details from early church history in completely incompatible ways.