Who is most responsible for the injustices that Tom Robinson and his family endure in To Kill a Mockingbird?

I would consider Bob Ewell as a direct antagonist to Tom Robinson. Bob rapes his own daughter and then blames Tom simply because Tom decided to help Mayella. An extension of Bob would be the white society of Maycomb. The community's inherent prejudice of blacks fuels Bob's baseless accusations of Tom.