routes started on the ends of the line or in the backfield, and which did not (and therefore could be ignored). It would complicate things too much for the defense to keep track of that through the duration of the play.

Plus, as shoganai pointed out, that would allow blockers to head downfield before the pass was thrown, which would be a big advantage to the offense (under the current rule, if you are going to have blockers downfield, the pass has to be caught behind the line of scrimmage).

I guess you could imagine the game without that rule, but it was decided very long ago that offenses would not be allowed to do that. I think these are some of the reasons why not.