Bayesian priors. Consider Indiana and Penn State. Indiana is undefeated with no top 25 wins and Penn State has lost to Ohio state. Why not rank IU higher? Based on our preseason knowledge, we had the prior belief that IU was worse than PSU for many reasons: Penn State had the 15th best recruiting class compared to Indiana's 65th best. Penn State has a stronger program that wins more most of the time, etc. etc. etc.
Anyway, if you have a strong prior belief that PSU is better than IU, it makes sense that the season so far will not convince you to switch their positions. IU could be worse and been lucky, while PSU could be better and been unlucky.
In fact, the case of IU/PSU is instructive. Consider that IU has not played tOSU yet. How likely are they to win that game? Aren't they basically on track to have the same record as Penn State, it's just that they haven't had their loss to Ohio State yet?
I think if you look past the bias in favor of the "blue bloods," you will find just a Bayesian prior that can be fully justified. For example Miami has the 4th best recruiting class and Texas has the 6th best. Miami is undefeated and Texas has a single loss to Georgia. Before the season started, you would have been right to believe they are both better than BYU (45th best class). Now, the records of these 3 teams has done nothing to undermine Texas and Miami's position. They have each won all their games just as you would expect a top team to do, except that Texas lost to Georgia, just as you would expect almost any team to do. "The eye test" is just a way to reward teams you know are better in cases where the records alone are not dispositive. The "blue bloods" that always seem to get the benefit of the doubt are also often the better teams: they have higher ranked recruits, more respected coaches and a better track record of performance. "The eye test" is just a way to take into account all the information that everyone knows.
(I hate "the eye test," also, fwiw. )