"When we forced him to retire, we took all the challenge out of his life."