The novel primarily takes place in New York during the 1950s. In the first section, we meet Rufus, a talented young Black jazz musician who is roaming the streets of the city, destitute and despondent. He reflects on recent events: he met a white woman from the South, newly arrived in New York, named Leona. The woman has had a difficult time of it, for her husband left her, taking their child. Rufus and Leona fell in love and decided to live together. But even the relatively tolerant milieu of Greenwich Village was unbearable for them. Rufus sharply felt that the world around him was hostile to their interracial relationship. Internal anxiety made Rufus look for occasions for quarrels with Leona; tides of passion alternate with acute alienation, when Rufus insulted Leona and even beat her. From the grief Leona became mentally unwell, and she was placed in a psychiatric hospital, where her brother visited and took her home to the South. Rufus devolved from a renowned drummer to a drunkard, and for this reason lost his job, and began to wander the streets of New York in torment. Weary from fatigue and hunger, he visits his friend Vivaldo, an aspiring writer, but even the sincere friendship of the latter, who all this time was looking for Rufus, does not save Rufus from unbearable loneliness, and he commits suicide by jumping from the George Washington Bridge.
Those around Rufus react differently to his death. Richard Silenski, a white writer who chased after commercial success and thus buried his talent, believes that Rufus himself is to blame for what happened. His wife Cass, a clever and strong white woman who always admired the talent and soulful qualities of the musician, believes that they, his friends, could have done more for Rufus—he should have been saved. So thinks Ida, the sister of Rufus: if her brother were with his family, among Black people, he would not have died. The trouble with Rufus, she believes, is that he was too sensitive and did not know how to defend himself. At Rufus’s funeral, to which Vivaldo and Cass come, the minister says that Rufus had lost his reason, left home, and stopped going to church. He encourages the mourners not to castigate Rufus for committing suicide.
The yearning for the departed Rufus brings Vivaldo and Ida closer together. They begin dating and move in together. Vivaldo continues working on his novel and Ida pursues a singing career. Vivaldo brings Ida into his circle of friends. Richard celebrates the release of his book. Vivaldo sees Richard, who was his high school teacher, as a mentor and envies his success (though after reading the book, Vivaldo is disappointed, for it is very pedestrian).
Richard also has new friends—not the poor bohemian Greenwich Villagers, but big publishers, literary agents, and bosses of show business and television. Once on a visit to Cass and Richard, Vivaldo and Ida become acquainted with Steve Ellis, a major television producer. He is struck by the beauty of Ida, and proclaims that if she has a talent in addition, he will help her advance. This makes Vivaldo very jealous, and the two begin the first of many tumultuous fights.
At this time Eric Jones, an actor, is about to return to New York from Paris because he was invited to play a role in a Broadway production. He is bisexual and several years ago fled from New York to escape from his passion for Rufus. The complexities of Eric's sexuality are rooted in childhood, spent in the South. The indifference and sometimes intolerance of his parents made Eric insecure and desirous for love. The only person who was kind to him was Henry, a Black man who worked in the boiler room. Eric’s first sexual encounter was with a Black teenager, and he eventually connects his feelings for Rufus to his experiences with these men. In Paris, Eric finally gained self-confidence, and he was no longer tormented by the thought of his "peculiarity.” In his art, Eric does not tolerate compromises; he is extremely demanding of himself and achieves a lot in his work. The thing that plagues him, though, is leaving behind his Parisian boyfriend, Yves, with whom he has found deep and abiding love and acceptance. Yves says he will come to New York in the near future, which thrills Eric but also makes him wary, for he hopes that their relationship can withstand the pressures of New York.
When Eric returns home and visits the Silenskis, the sensitive Cass instantly observes the difference between the former Eric and the one who returned to them after years of separation. Eric mercilessly analyzing himself and his actions is quite unlike Richard, or rather, the person that her husband became. In Richard there appears the self-assurance of mediocrity. Cass is now deeply disappointed in her husband, and wonders where the old Richard is and if his success was worth it.
Between Cass and Richard a divide is growing. Cass does not speak openly about her discontent; she is locked in herself and her husband notices her distance but seems unaware of larger issues. Now Cass stays outside in the city for a long time, for being at home for her is a torture. During one of these walks, she visits Eric. They begin an affair, and though they know it is temporary, they both feel an irresistible need for warmth and support from the other. In the meantime, Ida gives her first performance in a small bar in Greenwich Village. The audience accepts the untutored young singer despite the lack of technique because she embellishes all this with an inimitable individual manner—a mysterious property that does not have a name. At the same time, Vivaldo learns that Ellis secretly supports the girl, paying for her lessons with a well-known teacher. The young man is not sure of anything, but knowing such people as Ellis, he realizes that they never do nothing for nothing. He is jealous, suffers, and starts throwing himself into his novel.
Crises within both pairs are resolved almost simultaneously. When Cass comes home late one night, Richard forces her into a frank conversation, and Cass tells everything as it is: about her doubts about their marriage, and about the relationship with Eric. Richard is profoundly wounded and blames Cass for the dissolution of their marriage. It seems like he may want to save it, but Cass herself is ambivalent. The fate of their union lies unknown at the end of the novel.
Ida confesses to Vivaldo that she has been sleeping with Ellis to get ahead in her career, and she explains to Vivaldo with a stony face what it means to be a Black girl in a world dominated by white men. When Rufus committed suicide, Ida decided that she would not go his way, but would resist the world and get everything she wanted from it in any way it happened. When Ellis appeared, Ida realized that after the affair with him, if she behaved cleverly, she would find something in herself. After parting with Ellis she returned to Vivaldo, hating and despising herself but unyielding in her assertion of the difficulties of white people ever understanding Black people. Vivaldo is gobsmacked by this confession because he passionately loves Ida. It is implied they will work it out, but it will not be easy.
In the last chapter, Yves arrives in New York and an ebullient Eric is waiting to greet him.
