Why were the country people afraid to come to the Inn?

The people were afraid of the strangers, who were thought to be smugglers.

The more we told of our troubles, the more--man, woman, and child-- they clung to the shelter of their houses. The name of Captain Flint, though it was strange to me, was well enough known to some there and carried a great weight of terror. Some of the men who had been to field-work on the far side of the Admiral Benbow remembered, besides, to have seen several strangers on the road, and taking them to be smugglers, to have bolted away; and one at least had seen a little lugger in what we called Kitt's Hole. For that matter, anyone who was a comrade of the captain's was enough to frighten them to death. And the short and the long of the matter was, that while we could get several who were willing enough to ride to Dr. Livesey's, which lay in another direction, not one would help us to defend the inn.