"She insisted that I couldn't really go to an inaugural ball"