My wife and I play a "go football!" game. Whoever can spot the most fans wearing unrelated jerseys of the same sport wins. It's a way to mock the idiocy.
When we go but are not fans of either team, we wear neutral colors. I would do the same if I ever went to a yewt game where BYU wasn't playing.