Why do the men keep away from Huck's raft?

On his way to shore, Huck meets two white men searching for runaway slaves. The men ask him who else is on his raft and rather than telling them about Jim, Huck tells them his Pa, mother, and sister are aboard. Huck pretends to be eager for their help and tells them no one else has been willing to pull the raft to shore. At this news, the men become suspicious and finally conclude that Huck's family must have smallpox.