Answer
a. $x'^2+3y'^2=9$
b. $\frac{x'^2}{9}+\frac{y'^2}{3}=1$
c. see graph
Work Step by Step
a. Step 1. Given
$6x^{2}-6xy+14y^{2}-45=0$
we have
$A=6, B=-6, C=14$
and the rotation angle is
$cot(2\theta)=\frac{A-C}{B}=\frac{6-14}{-6}=\frac{4}{3}$,
Step 2. We have the transformations (see also section 9.4 exercise 16):
$x=\frac{\sqrt 10}{10}(3x'-y')$ and $y=\frac{\sqrt 10}{10}(x'+3y')$
Step 3. The equation becomes
$6x^{2}-6xy+14y^{2}-45=6(\frac{\sqrt 10}{10}(3x'-y'))^{2}-6(\frac{\sqrt 10}{10}(3x'-y'))(\frac{\sqrt 10}{10}(x'+3y'))+14(\frac{\sqrt 10}{10}(x'+3y'))^{2}-45=5x'^2+15y'^2-45=0$
or, $x'^2+3y'^2=9$
b. In standard form, we have $\frac{x'^2}{9}+\frac{y'^2}{3}=1$ which represents an ellipse.
c. We can graph the equation as shown in the figure
