I decided to do a deep dive on how well calibrated the ESPN FPI model is. If you're not familiar with this, it's just a means to test how often a probabilistic model predicting a binary outcome is correct. For example you could look at how often teams with a 70 percent chance of winning actually win. If it's 70 percent then your model is doing a good job, at least for predictions in that range.

The results are below. I pulled FPI predictions and win results for the last eight seasons or so, including this season to date. The FPI prediction is the win probability of the home team. I created buckets of the ESPN FPI predictions and then looked at the actual win percentage of the teams in those buckets. So the 0-10 bucket means the ESPN FPI prediction was somewhere between 0 and 10%. Looking all the data, teams in this range won about 7% of the time. A perfectly calibrated model probably should be hitting around the midpoint of each of these buckets.

I decided to split it out and look at the calibration for the early season versus the later season. I thought that the model might improve later in the year with more information about the teams. However, I think the challenge of predicting games later in the season offsets whatever model improvements there are. Looking at some statistical measures I'm not showing here, the model actually does worse in the later season. But I think this is mostly because there are fewer freebie games later in the year (e.g. BYU vs SIU). 

In both the early and later parts of the season, the model struggles the most with predicting the home team when the home team is an underdog. It seems to overestimate the home team early in the season and underestimates the home team later in the season.

As far as this weekend, BYU has a 67.7 win probability. Based off the last several seasons, teams with this win probability went on to actually win right around 66 percent of the time. It still feels like FPI might be under valuing BYU, but over thousands of games, it has been a pretty decent indicator of what to expect.