"There were always whispers"