Answer
$(0,\pm1)$
Work Step by Step
Step 1. Given
$5x^{2}-6xy+5y^{2}-8=0$
we have
$A=5, B=-6, C=5$
and the rotation angle is
$cot(2\theta)=\frac{A-C}{B}=\frac{5-5}{-6}=0, 2\theta=\frac{\pi}{2}, \theta=\frac{\pi}{4}$
Step 2. We have the transformations (see also exercise 7):
$x=\frac{\sqrt 2}{2}(x'-y')$ and $y=\frac{\sqrt 2}{2}(x'+y')$
Step 3. The equation becomes
$5x^{2}-6xy+5y^{2}-8=5(\frac{\sqrt 2}{2}(x'-y'))^{2}-6(\frac{\sqrt 2}{2}(x'-y'))(\frac{\sqrt 2}{2}(x'+y'))+5(\frac{\sqrt 2}{2}(x'+y'))^{2}-8=2x'^2+8y'^2-8=0$
or, $x'^2+4y'^2=4$
Step 4. In standard form, we have $\frac{x'^2}{4}+\frac{y'^2}{1}=1$, which represents an ellipse.
Step 5. We can find the endpoints of the minor axis in the $x'-y'$ system as $(0,\pm1)$
