Growing up as a non-member yewt, I can confirm this. Once I joined the Church,
I couldn't take the cognitive dissonance of hating BYU. I can't speak for everyone, as certainly there are lots of reasonable non-members (most) in Utah, but I was definitely in the subset that was more divisive.