"Otherwise, the next thing you know, Downing Street will overreact,"