A full season is hard to not have any off games and team lose a few that they should, or would typically win. This is especially true in basketball (look at BYU in the WCC).
For OU and Texas, now in the SEC, when they lose that inevitable game, they get a pass becuase it was to an "SEC" team. And the fan base can tolerate much more losing to Georgia than to BYU, or Kansas, or Utah, or UCF.
Agree completely that the difference in diffucutly of the schedule (while a factor), isn't as large as the difference in perception.