If your goal is to win a conf championship, then out of conference games don’t interfere with that goal, win or lose. But playing marquee matchups help you maximize revenue and exposure.
If your goal is to simply win 6 games and go to the toilet bowl every year, with no thought of ever building depth and improving…then sure, I guess you’re right and our schedules are hard enough already.