What you describe is very close to how I envision our lifestyle living there. The run down parts of town really don't bother me. It's endearing in a way.
Life just feels a little slower there which I can 100% get on board with. We have a couple of homes on our radar we will be checking out in the next week or so.