Narrated through voices and points of view of several characters, The Longest Memory begins with Whitechapel, the novel's protagonist. An aging and obedient slave living on a Virginia plantation in the early 1800s, Whitechapel opens the novel with an elegiac monologue about the pain of remembering. Whitechapel renounces his name as part of his effort to stop remembering the past.
Whitechapel says he has seen too many deaths for one lifetime, but his son Chapel's death was most traumatizing. He watched the slave overseer's whip bite into Chapel's back two hundred times. He watched Chapel dissociate from the pain, and from then on Whitechapel himself took on the same faraway look. Whitechapel explains that Chapel tried to escape the plantation the day after Chapel's mother died. Whitechapel told the plantation owner, Mr. Whitechapel, which route Chapel took because Whitechapel believed his son needed to be taught a lesson in obedience and drive any notion of freedom from Chapel's head. Because Chapel died from the two hundred lashes that were his punishment for trying to escape, Whitechapel's descendants on the plantation have held the old man responsible for Chapel's death.
The narration shifts to Mr. Whitechapel's monologue on the day he returns to the plantation to discover that his overseer, Sanders Junior, whipped Chapel to death. Mr. Whitechapel reprimands the overseer for not listening to Whitechapel, with whom Mr. Whitechapel had left instructions to detain but not harm Chapel until Mr. Whitechapel returned from the North with his son, daughter, and wife. Mr. Whitechapel disapproves of Sanders' rough handling of the slaves and says that despite their racial inferiority Africans exhibit the same qualities as white people. Mr. Whitechapel informs Sanders that his father, Sanders Senior, in fact raped Chapel's mother, and so Sanders whipped his own half-brother to death.
The narration shifts next to Sanders Senior's diary from the late 1700s. The diary gives an account of Sanders' life on the plantation as the overseer of the slaves. His wife Caroline died in childbirth, and Sanders cannot tell his five-year-old son how the boy's mother died. Sanders details how he hires a new cook, whom he dubs Cook, and how Whitechapel takes a romantic interest in her. Sanders Senior struggles with his loneliness and reprimands himself for fantasizing about Cook, and eventually rapes her. Cook marries Whitechapel. A week later Sanders Senior rapes her again. He wakes the next morning to Cook, Whitechapel, and Mr. Whitechapel standing in his room. Negotiations follow, and Whitechapel says he will raise Cook's son as his own. Mr. Whitechapel arranges for Sanders to marry a woman to whom Sanders isn't attracted as a condition for staying on the plantation. Mr. Whitechapel swears everyone to secrecy about Chapel's conception.
In Cook's narration, the reader learns of Cook's perspective on the account Sanders' diary has put forward. Cook says she wanted to die after he laid his hands on her, and she is grateful that Whitechapel didn't reject her after learning of the rape. She vows to bear Whitechapel many sons.
The narration switches to Chapel's voice. In rhyming couplets, he speaks of how he learns to read and write from Lydia, Mr. Whitechapel's daughter. After Mr. Whitechapel catches Chapel reading Shakespeare aloud to Lydia, Mr. Whitechapel beats Chapel and forbids him from seeing Lydia. Keeping his promise not to see her, Chapel begins meeting Lydia outside at night in the dark; they sit on a bench, back to back. She memorizes literature which she recites to Chapel. Over time, they decide to escape to the more socially liberal North, where they may live openly as a couple. Chapel's account ends with him talking to his father before he escapes. He dismisses the old man's advice not to seek freedom with a shake of his head. He sets off running, feeling joy, not fear.
The narration shifts to Mr. Whitechapel's internal monologue as he visits a social club for plantation owners. The fellow slave-owners mock him for the hypocrisy of preaching lenient treatment of slaves while allowing one of his overseers to beat a slave to death. Mr. Whitechapel feels shame, but he succeeds in winning the slave-owners over when he tells them that Whitechapel didn't tell Sanders Junior that he was beating his own half-brother to death. Impressed with Whitechapel's extreme deference to authority, the men raise a toast to Mr. Whitechapel and his slave.
Lydia's voice briefly takes over the narrative as she explains how she teaches Chapel how to read and write when they are children. The voice returns to Cook as she recounts how she catches Chapel reading aloud to Lydia but cannot summon disdain and punish him because of the pride she feels at his intelligence and at the strength of his voice. Lydia then recounts how she falls in love with Chapel's reading voice. Her father tells her off when he learns that she taught Chapel to read. Lydia wants to say it is unjust to disallow a slave from reading and writing, but she merely looks at her feet. Lydia says Cook told her about a good spot to view the stars with no one else around. Lydia goes out to the bench and begins meeting Chapel in the dark. With tears in their eyes, she and Chapel say that they love each other.
After years of meeting Chapel in the dark, Lydia grows into womanhood and her parents send suitors her way. She hates the suitors and wishes she could be with Chapel; she wishes he were white or she were black. Her brother travels North for business and tells Lydia about how free black men walk the streets of Boston and New York with white women on their arm. She imagines herself doing the same. Lydia and Chapel make plans to meet in the North: she will travel with her family and then escape from them, while Chapel will run from the plantation. To support their family, Chapel will work as a poet.
The narration shifts to a series of newspaper editorials printed in The Virginian. Written by an anonymous slave owner, the editorials cover issues around the treatment of slaves, the economics of the slave trade, legal issues, what to do with a runaway, whether Christianity is compatible with the slave trade, and whether the paper should include accounts of slaves. One of the editorials alludes to Chapel's death. Another speaks of Miss L., a likely pen name for Lydia, who writes in to say literacy should be available to all and that the editorial's arguments against black men being with white women are unconstitutional.
The narration switches to the voice of one of Whitechapel's great-granddaughters as she recounts Whitechapel bathing her as a child. When she wants to know about his boyhood in Africa and shares her own dream of flying to Africa and kissing the soil, the great-granddaughter is discouraged from dreaming about something she doesn't know. She says Chapel's death deadened Whitechapel's eyes. Everyone treats him like a ghost they all see but ignore because he killed his only son. She dreams of the whip and Chapel's own deadened, detached eyes.
The narration switches to Sanders Junior's internal monologue as encounters Whitechapel's dead body on the ground. Sanders tells Whitechapel that he was a good slave who always knew his place; he'll get a grave with a stone. Sanders covers him with his jacket, which belonged to his father. Sanders talks of his father, whom he believes to be more cowardly than Whitechapel. Sanders says he would have preferred Whitechapel as his father, if only Whitechapel had been white.
The novel ends with an epilogue presenting Whitechapel's internal monologue as he dies. Whitechapel speaks of how Chapel's happiness wasn't important to him, only Chapel's life. He told him what he knew as a slave, which was not to seek freedom to save his life. He raised Chapel as a slave, unaware of his mixed blood, but he believes Chapel was following the dictates of his dreams and listening to his blood when he sought freedom. Whitechapel reflects that he has been wrong all his life. He lies on the ground to rest, longing to forget his painful memories.
