The first thing that crosses his mind is 'no I didn't,' because he can't have. Until he remembers that apparently he was back in Neverland, it just hasn't happened yet.

"I left the ship, and I couldn't leave Neverland itself so I had to stay on the island. I refused to be one of the lost boys, so that left the cave."