Answer
At such a depth, the water pressure is tremendous. A larger viewing window would have a correspondingly larger force on it, possibly cracking the window. A small circular viewing window makes sense - with no corners, there are no points of stress concentration.
Also, such windows are often tapered, like a cone with the sharp end cut off. By placing the larger circle on the outside of the sub, the water pressure helps press the window into its frame, making a tight seal.
