I can understand going to a game between 2 good teams, even if one is your rival
However, I would never wear my teams' colors while at that game.
It's one thing to go to a good game between two teams for the entertainment value. It's very weird to go to your rival's game against some random school and sport your team's colors.