It's essentially a third team out there who could be having a good or bad game too. Their calls have an incredible amount of influence over how both teams play.

It's one of the reasons I don't really enjoy basketball as much as I do football. Too much influence. Football has also gotten a lot worse due to all the highly subjective rules that have been added (targeting, substitutions etc.)