"If I'd gone, it would have destroyed him. I couldn't do it."