It is the year 1962 in Hong Kong, thirteen years after the establishment of the People's Republic of China under the leadership of Mao Tse-tung, the same year Mao proposed the Socialist Education Movement that would be a major precursor to the Chinese Cultural Revolution. At this time, Hong Kong is seeing an influx in migration of mainlanders to the island, particularly those wary of political upheaval. The film begins in the bustling apartment of Mrs. Suen, who is engaged in a spirited conversation with her neighbors and mahjong partners, the Koos. A young married woman named Su comes to Mrs. Suen about the room she has listed for rent in her apartment. Su is inquiring on behalf of herself and her husband, who frequently travels for work. They're looking for a place to live in Hong Kong. Mrs. Suen rents Su the room.
As Su is leaving, a young man, about her age, comes to Mrs. Suen's door. He's inquiring about the same classified ad. His name is Chow. Mrs. Suen tells him that she just rented the room, but that he should speak to Mr. Koo next door; he has a room to rent, because his son has just gotten married and moved out of the apartment. Shortly after this encounter, Chow and Su move into their respective rooms. Move-in is made even more hectic and claustrophobic due to both couples moving in on the same day, at the same time. Neither Chow's wife nor Su's husband is present for the move-in. The movers keep accidentally switching items, bringing Chow's things to Su and vice-versa.
Once they're all moved in, Chow and Su encounter one another in the alley and by the noodle stalls, where they both often get dinner for themselves, in the (frequent) event that their spouses are working late. Life goes on as usual for a time—Su works as a secretary to a businessman named Mr. Ho; Chow works as a reporter for a Hong Kong newspaper. The Suens and Koos are dazzled by the things Su's husband sends and brings home from his travels in Japan, particularly an electronic rice cooker. He brings one back for Mrs. Suen and Chow. He also brings back fine gifts for Su's boss, Mr. Ho. Specifically, the gifts are for Mr. Ho's wife and mistress. Su logistically facilitates Mr. Ho's affair by scheduling dates with his wife and mistress, making sure he has gifts for them on special occasions, and making sure the wires never cross, that both women are happy, and that Mrs. Ho never hears about his mistress.
As the weeks go by, it becomes clear that Chow's wife is lying about where she is. One night, when she tells him not to wait up, that she'll be late home from work, Chow shows up at her workplace to take her out to dinner. Another person who works there tells Chow that she left early that day. Then, one afternoon, Su hears her husband's voice coming from the Suens' apartment. She knocks on the door, and Chow's wife answers. Chow's wife claims to be home alone after calling in sick from work. Su leaves the apartment, but cries to herself in the shower because she knows her husband is having an affair. Soon after Su's husband leaves for Japan, Chow's wife also leaves for Japan, supposedly also for work purposes.
One night, Chow asks Su to join him for dinner at a nearby diner. He asks her out on the pretense of wanting to know where she got her handbag so that he can purchase his wife one for her birthday. Su tells him that she'd have to ask her husband, since he got it for her in Japan. They don't sell them in Hong Kong. Then Su asks Chow where he got his tie, and he tells her the same thing she told him about the bag—his wife got it for him in Japan. Su admits that her husband has a tie just like it, and Chow admits that his wife already has the same type of handbag that Su has. They both recognize that their spouses are having an affair with one another.
After they leave the restaurant, they start to theorize how the affair began. Then they decide to start roleplaying their spouses, in order to play through some possible scenarios. They start to see each other more often, meeting at the restaurant, each pretending to be the other's spouse. In the course of doing this, they become close companions. Chow shares with Su his dream of writing a martial arts serial for newspapers, and she encourages him to do it. They end up working together on a serial in their spare time. But as they spend more time together—covertly, so as not to provoke rumors—Mrs. Suen takes notice of Su's general absence from the house. She lectures Su about being out too much and not spending enough time with her husband. She suggests that Su should try to prevent him from traveling so much in the future. After being shamed and lectured, Su pulls away from Chow, but Chow doubles down and gets an apartment in another building where they can work together in peace, without the prying eyes of their neighbors. At first, Su rejects the idea, but eventually she visits him at his new place, and they continue work on the serial.
But after a while, Chow can no longer deny his feelings for Su. Recognizing that she will never leave her husband to be with him, he asks Su if she would set him up with a ticket to Singapore. His colleague at the newspaper, Ping, told him that the Singapore Daily is short-staffed and looking for reporters. Su asks him why he's rushing off to Singapore, and he admits that while it was never his intention to fall in love with her, he has, and that if she is unable to be with him, there's really nothing keeping him in Hong Kong. He says he needs a change of scene. But before he leaves, he asks her to rehearse their final goodbye, just as they rehearsed their confrontations of their spouses. While rehearsing, Su weeps into Chow's arms. It's clear at this juncture that their relationship has transformed into something genuine, far more than the roleplay as which it began.
Three years pass by after Chow leaves for Singapore. Su visits Mrs. Suen's apartment in Hong Kong and finds Mrs. Suen in the middle of packing the place up. She's leaving for the United States to help her daughter raise her children. Suen admits that she doesn't want to leave Hong Kong, but her daughter is bothered by the political situation there, and that if she likes it in the U.S., she may never return. Su expresses interest in renting the apartment. She then asks if the Koos still live next door, and she holds back tears when Mrs. Suen reminisces about the times past, clearly thinking of her time spent with Chow.
Some time later, Chow visits the Koos' old apartment to see if they still live there, but the man who answers the door tells him that they've been gone for a while. He says that everyone is leaving to escape the chaos of Hong Kong. When Chow asks whether Mrs. Suen is still in the apartment next door, the man says the same thing, but tells him that a young woman lives there with her son. Chow looks through the window and smiles, understanding that his relationship with Su is something that permanently lives in the past.
The film then cuts to documentary footage of Charles de Gaulle visiting Cambodia, being greeted by the royal family at the time, calm and order just before a long period of unrest and war in Cambodia. The scene then moves to the temple, Angkor Wat, where Chow whispers a secret into a small hole in the temple wall. Afterward, he fills the hole with soil, enacting the ritual of burying secrets that he described to Ping in a restaurant in 1963, three years earlier. The film ends with sweeping shots of the vacant temple and a title card ruminating on the end of eras and times past.
