"But he's got his own family. I need to focus on mine."