Most of my adult BYU fan friends have retired from Mormonism but wear as much as they ever have. More so even because they've dropped the tendency to care what other people think of them. LDS BYU fans are sometimes quite worried about "getting crap" from other fan bases. Retired Mos, for the most part, don't.
The other factor that I think contributes to the lack of sales is that our fanbase is rather cheap. Big families and tithing means not a lot of extra money for the latest colors.