Answer
$(0,\pm\sqrt 6)$
Work Step by Step
Step 1. Given $2x^{2}-4xy+5y^{2}-36=0$, we have
$A=2, B=-4, C=5$
and the rotation angle is
$cot(2\theta)=\frac{A-C}{B}=\frac{2-5}{-4}=\frac{3}{4}$
Step 2. We have the transformations (see also exercise 15):
$x=\frac{\sqrt 5}{5}(2x'-y')$ and $y=\frac{\sqrt 5}{5}(x'+2y')$
Step 3. The equation becomes
$2x^{2}-4xy+5y^{2}-36=2(\frac{\sqrt 5}{5}(2x'-y'))^{2}-4(\frac{\sqrt 5}{5}(2x'-y'))(\frac{\sqrt 5}{5}(x'+2y'))+5(\frac{\sqrt 5}{5}(x'+2y'))^{2}-36=x'^2+6y'^2-36=0$
or, $x'^2+6y'^2=36$
Step 4. In standard form, we have $\frac{x'^2}{36}+\frac{y'^2}{6}=1$, which represents an ellipse.
Step 5. We can find the endpoints of the minor axis in the $x'-y'$ system as $(0,\pm\sqrt 6)$
