Given that I was planning to make extra payments anyways, it made sense to refinance with 22 years left on my 30 year mortgage that had a 3.5% rate. At that point, going with a 15 year felt like the right move for me personally.
I know, just based on numbers, what you're saying is true. But, having your house paid off and no monthly mortgage payment to make provides a certain level of comfort for me.