Well, no one knows the physical condition of Amelia and Fred when they arrived on the island. But for sure, they needed water, food and shelter (since sleeping on the bare ground must have been next to impossible with those crabs). Assuming that they KNEW a shipwreck was there on the island, would they not think it worth taking a look to see what they could scrounge around for items they could use to survive.
What I am getting at is that perhaps they also spent some time there at the Norwich survivor site. Although that area must have been visited and even picked through over the years, it might be worth exploring as well when the expedition goes out there in July.