Told from the point of view of a young boy named Billy, The Giraffe and the Pelly and Me tells the story of Billy's lifelong dream to own a candy store (which the British call a "sweet shop.") He almost constantly is reminded of this dream because of an abandoned building next to his home called the Grubber (an old term for a candy store).
But his dream of renovating the building is dashed when one day he notices that the building has been renovated for The Ladder less Window-Cleaning Company. The company, in fact, is staffed by strange but exciting animals whom Billy quickly becomes friends with.
Billy and the animals are one day asked by the Duke of Hampshire to clean the titular Hampshire house. They accept the Duke's request. As they clean one day, however, they catch a cat burglar trying to steal the Duke's wife’s jewels. Because of this, the Duke invites the animals to live on his estate so that they can be his personal helpers, meaning that they no longer their building. In turn, Billy turns their old building into a candy shop, fulfilling his lifelong dream.