Answer
a. $3x'^2+y'^2=20$
b. $\frac{3x'^2}{20}+\frac{y'^2}{20}=1$
c. see graph
Work Step by Step
a. Step 1. Given $x^2+xy+y^2-10=0$, we have $A=B=C=1$.
The rotation angle is
$cot(2\theta)=\frac{A-C}{B}=0, 2\theta=\frac{\pi}{2}, \theta=\frac{\pi}{4}$
Step 2. We have the transformations:
$x=\frac{\sqrt 2}{2}(x'-y')$ and $y=\frac{\sqrt 2}{2}(x'+y')$
Step 3. The equation becomes
$x^2+xy+y^2-10=(\frac{\sqrt 2}{2}(x'-y'))^2+(\frac{\sqrt 2}{2}(x'-y'))(\frac{\sqrt 2}{2}(x'+y'))+(\frac{\sqrt 2}{2}(x'+y'))^2-10=\frac{3}{2}x'^2+\frac{1}{2}y'^210=0$
or
$3x'^2+y'^2=20$
b. In standard form, we have
$\frac{3x'^2}{20}+\frac{y'^2}{20}=1$
which represents an ellipse.
c. We can graph the equation as shown in the figure.
