"If I had Brancaster Castle, I'm not sure I'd ever want to leave."