My grandfather (friends with Thomas S Monson) had a similar question about leaving a good job to help build a new location and wasn't sure what to do. Elder Monson questioned "How can the Lord move you if you won't go?"
A situation like this seems like a good opportunity to make a pro and con list and get to the Temple.