While not really accurate, the thought is Jews are somewhat nerds — more academic than athletic. Probably more importantly is the issue that most high school and college games tend to take place during the Jewish Sabbath (Friday evening until Saturday evening).
By contrast, we have basketball courts/gyms in all of our church buildings, as well as high school and college games rarely being played on Sunday.
It likely also helps that a large percentage of Jewish people (about 40%) live in the Northeast, where football is less popular, whereas most Mormons in the US tend to live in Western states.