That's where I think they could say it isn't a defenseless player at that point because he clearly adjusted to it; so as long as there isn't leading with the crow by the defender to trip the first set of tests, targeting under the second set could be ruled out and no foul called. Often it does get called in those instances, which makes it so the more frustrating.
It is so broad and nebulous that the reason it either is or isn't called in a particular instance should be spelled out.