How is Crooks isolated?

Crooks is isolated to the horse stables because he is black:

S'pose you didn't have nobody. S'pose you couldn't go into the bunk house and play rummy 'cause you was black. How'd you like that? S'pose you had to sit out here an' read books. Sure you could play horseshoes till it got dark, but then you got to read books. Books ain't no good. A guy needs somebody-to be near him. A guy goes nuts if he ain't got nobody. Don't make no difference who the guy is, long's he's with you. I tell ya, I tell ya a guy gets too lonely an' he gets sick"