Why is Alfred Doolittle so unhappy to be wealthy?

Alfred Doolittle is angry that Higgins for giving his name to Ezra D. Wannafeller. Higgins had espoused Doolittle's high morals, perhaps as a joke, and Ezra D. Wannafeller left money to Doolittle. Doolittle hates being part of "middle class morality." He wishes to be free and poor rather than having money and forced to face family and friends who want it.