How does the party control the past? How might this be terrifying?

The party simply changes the past. They re-write history. People who know the truth die off and the re-written history becomes "true". This is pretty terrifying because truth and history become malleable. Unfortunately, this is happening even today.