Lee specifically states that the rumors came from what people "said", not what they saw.
People said he existed, but Jem and I had never seen him. People said he went out at night when the moon was down, and peeped in windows. When people’s azaleas froze in a cold snap, it was because he had breathed on them. Any stealthy small crimes committed in Maycomb were his work. Once the town was terrorized by a series of morbid nocturnal events: people’s chickens and household pets were found mutilated; although the culprit was Crazy Addie, who eventually drowned himself in Barker’s Eddy, people still looked at the Radley Place, unwilling to discard their initial suspicions. A Negro would not pass the Radley Place at night, he would cut across to the sidewalk opposite and whistle as he walked. The Maycomb school grounds adjoined the back of the Radley lot; from the Radley chickenyard tall pecan trees shook their fruit into the schoolyard, but the nuts lay untouched by the children: Radley pecans would kill you. A baseball hit into the Radley yard was a lost ball and no questions asked.
