I'm learning that ESPN, and apparently ranking services have a built in bias for the SEC and B1G conferences. They are ranked really high when recruited and playing for teams in those conferences...and so, if you have a bunch of 4 and 5 star players, then obviously...you're going to get the pre-season rankings that justifies all those star players.
As soon as they transfer out...they are downgraded in their rankings so now, teams like BYU and others don't warrant the hype in the preseason and are ranked way down the list.
That's how is kind of rigged. A team ranked in the top 5 losing to a top 10 team (because of the biased player rankings)...don't drop much in rankings at all when they lose. But, a BYU or other team in any other conference will drop significantly after a loss....you know, because they just can't be that good with all those 3 star players.
So...yeah. It kind of makes a difference in a phony way. But, it's true if the media and ranking committees believe it.