By the time you do a H/A with those teams, play 1 division in your conference
and 1 division on the opposing conference that leaves like 3 games. Every 4 years you get to play every team in the opposing conference. Maybe that's not frequent enough, but the scheduling to me seems to be the best they can do.