"'Why, we just, we were almost like an invisible town to people..."