He's had plenty of time to "put his mark" on the program. Instead he's been hiring/firing coaches for poor performance and has developed a reputation among fans of not being a able to gameplay well or recruit well.
Fans are not happy with the trend, but even his strongest detractors don't hate him. Just like they don't hate Ty Detmer, Brandon Doman or any other player that has also been a coach.
He is still part of the BYU family.