What misery does the character runs away from.

The speaker runs away from the sight of the castaways..... their loneliness and devastation, as it is something he doesn't mind knowing about.... but he doesn't want to see the reality.

The castaways of life, a few asleep,
Some withered women desolate and mean,
And over all, life’s shadows dark and deep.
Moaning I turned away, for misery
I have the strength to bear but not to see.