"We would swim to that island every day."