Why does Papa feel that “friendship between black and white” doesn’t mean that much?

Papa does not think it means much because the friendship can never be equal.

“Far as I'm concerned, friendship between black and white don't mean that much cause it usually ain't on an equal basis. Right now you and Jeremy might get along fine, but in a few years he'll think of himself as a man, but you'll probably still be a boy to him. And if he feels that way, he'll turn on you in a minute.”