If it’s important to you to get where you want to be, then just do it.
The alternative is to find a job that won’t ever lead to as prestigious of a career but will provide the life you want and bail like a smaller job in a normal town.
The firm could care less if you bail but we tend to get wrapped up into thinking about how important it is for us to stay. Spit the koolaid out and leave.