Eh, there’s a little more to pole vault than that. Pole length, stiffness, grip height, where to place the standards relative to the plant box. Then there’s wind. Those variable above can all change whether the wind is blowing in your face, from the left, the right, from your back. A slight gust of wind from the side when you have a 17’ pole dangling in front of you is like trying to hit a nasty curve ball. A variation of even 2-3” where your takeoff foot lands could be the difference between a great jump and a terrible one.
What you said is like saying batting is just see the ball and swing at it. it’s “muscle memory” at that point too.

That being said I know very successful pole vaulters who are terrible at other sports. And I know some who could have played multiple sports in college, but preferred pole vault.

The real answer might be elite (world class) decathletes who played baseball, basketball and football growing up.