and we are inside our own 20, we have the lead, tie game, or even down a FG. You are playing ball security and field position. You either run it or throw a very short but safe throw. The worst possible thing is turning it over in your own territory. You basically are gifting them points.
Next situation: Same field position, end of game scenario, we are down a FG or need a TD to tie or win. The defense knows that it is 4 down territory no matter what. Do I want to have a 4th and long play? Nope, the conversion % is extremely low. So while fans might pull their hair out on a 4-5 yard play on 3rd because we're not "going for the sticks," the question is, why force a long throw when 1) the defense is expecting and wanting that and 2) you have 2 more plays. 4th and 4-5 has a higher chance to convert than 4th and 10. It is also infinitely better than turning it over by forcing a throw.