How did Tom know the large man he met was neither Indian nor negro?

From the text:

He was exceedingly surprised, having neither heard nor seen any one approach; and he was still more perplexed on observing, as well as the gathering gloom would permit, that the stranger was neither negro nor Indian. It is true he was dressed in a rude Indian garb, and had a red belt or sash swathed round his body; but his face was neither black nor copper-color, but swarthy and dingy, and begrimed with soot, as if he had been accustomed to toil among fires and forges.