Drake is basing his opinion on actual performance on the field.
One of the most frustrating things about college football rankings is there aren't enough games or common opponents to determine fair rankings. As a result, the voters base their votes on all sorts of other metrics other than on-field performance.
The 12 team playoff is designed to fix this, and it will to some extent. But there will always be commentators that come up with lots of ways to rank teams other than on-field performance.