When the men recognize the voice of Ben Gunn, why aren’t they afraid anymore?

Initially, they believe the voice singing the song is that of Flint's ghost. They were relieved to find it was Ben Gunn, as he ws no threat.

At first everyone thought that it was the voice of Flint's ghost. Later, they realized that it was Ben Gunn who was singing and wasn't a threat to them.