Physicality has little to do with their struggles.
It all comes back to abysmal qb play and an outdated and unimaginative offense. They scored 7 points against tcu and 10 against arizona...neither of which could remotely be considered a physical defense. Teams aren't pushing them all over the field.