Who did the hunter think Phoenix was going to town to see?

During her journey, Phoenix has an altercation with a dog that comes after her. She hits the dog wih her cane, but falls over in turn. At this point, a white man—a hunter—helps her from the spill she took into the ditch. He starts out nicely by asking her if she all right and then asks where she is going. When she answers that she is headed into town, he laughs that “colored people” never want to miss “going to town to see Santa Claus.”