Speaker
The speaker of the poem is most likely a representation of Trethewey herself, as a number of her biographical details are written about in the poem. The speaker disagrees with her father's assessment of Thomas Jefferson's historical legacy, while also feeling that his defensiveness about the subject is informed by his connection to it, as the father of a mixed-race daughter.
Speaker's father
Given that the speaker of the poem is a version of Trethewey, it is reasonable to assume that the father described in the text also represents her real-life parent. He is also wrestling with the actions of Thomas Jefferson, but attempts to defend him in spite of his shortcomings. The speaker feels that some part of her father's argument is a product of his personal connection to the subject of mixed-race parentage.
Thomas Jefferson
The poem alludes to Thomas Jefferson's historical legacy, describing a famous portrait of him that hangs in his Virginia home. The speaker's main issue with Jefferson was the gap between his lofty ideas about freedom and his ownership of slaves.
Sally Hemings
The poem also makes reference to Sally Hemings, the slave with whom Jefferson was known to have had children. The speaker is disturbed by the fact that he could father children with a person he owned.