"You're very kind, Lady Grantham, but I must get back to London."