"There's a time when the joking has to stop though."