For this excercise I will use 120 FBS teams (although there are currently 128) and 350 CBB teams (although for much of the Dave Rose era there were less than that).
The BCS had 10 spots, but 6 spots BYU cannot claim. They were guaranteed to the BCS conference champions. Any team can claim any of the other spots. So in reality, BYU had a 4/114 chance of making the BCS. That works out to about 3.5%.
To make the Sweet 16 you have to make the tournament. You can make the tournament automatically as a conference champion or as an at-large.
In the Dave Rose Era, to automatically make the tournament and be the conference champion they would have to win 3 games in a row. Assuming each game is a 50/50 proposition that means BYU has a 12.5% chance of being a conference champion.
There are 33 automatic bids for conference champs which means there are 31 at-large bids to automatically go to the 2nd round. 31/317= 9.7%. I'm not even counting the play-in game option here.
So BYU has a 22.2% (12.5% conference champ route plus 9.7% for at large) chance to put themself in a position to win 2 games to make it to the sweet 16. Each game there is a 50/50 chance that they win. 22.2%*50%*50%= 5.55% chance for BYU to make it to the Sweet 16