"He's their grandfather. It's not like he's a dog"