"I figured it was because he didn't want to be around any of it,"