Why does Jem think Boo Radley wants to stay in the house?

Scout thinks that people are the same. Jem counters that there are different kinds of people:

"If there's just one kind of folks, why can't they get along with each other?  If they're all alike, why do they go out of their way to despise each other?  Scout, I'm beginning to understand something.  I think I'm beginning to understand why Boo Radley's stayed shut up in the house all this time...it's because he wants to stay inside."

Jem is beginning to realize why Boo chooses to stay inside. It is other people that have the problems, not Boo.