"I'd leave, Caroline would feel neglected,"