He would've known JoJo made a cut move to beat him. He reached back out to hold, choosing to risk giving up 10 yards on the penalty rather than a much bigger chunk play and potentially even a touchdown to BYU. While the 10 yard penalty gave BYU new life to extend the drive with a new set of downs, it also kept BYU deep in their own territory.
It all looks like a major mistake and failure on Vaughn's part because of how it seemingly backfired on him and his whole team with how BYU was able to capitalize on the extra set of downs. But seeing how JoJo was probably the primary target on a hot route for that play call coming out of BYU's timeout, the choice to hold and accept the risk of getting caught makes some sense.
So I can understand why Whit would coach a player to do that and rely on their defense to stand their ground through another set of downs deeper in the opponents territory rather than risk giving up the bigger gain that would also result a new set of downs while allowing the opponent closer to their goal of getting into FG range all while leaving more time on the clock for the opposing offense to work with. Or worse yet, allowing the opponent to immediately get into FG range or score a TD on that play.
In other words, while it was risky to commit the hold, it was probably more risky to not commit that hold and let JoJo have free run on his route in that situation.