Answer
a. $26x'^2+y'^2=9$
b. $\frac{x'^2}{9/26}+\frac{y'^2}{9}=1$
c. see graph
Work Step by Step
a. Step 1. Given
$10x^{2}+24xy+17y^{2}-9=0$
we have
$A=10, B=24, C=17$
The rotation angle
$cot(2\theta)=\frac{A-C}{B}=\frac{10-17}{24}=-\frac{7}{24}$
Step 2. We have the transformations (see also exercise 21):
$x=\frac{1}{5}(3x'-4y')$ and $y=\frac{1}{5}(4x'+3y')$
Step 3. The equation becomes
$10x^{2}+24xy+17y^{2}-9=10(\frac{1}{5}(3x'-4y'))^{2}+24(\frac{1}{5}(3x'-4y'))(\frac{1}{5}(4x'+3y'))+17(\frac{1}{5}(4x'+3y'))^{2}-9=26x'^2+y'^2-9=0$
or
$26x'^2+y'^2=9$
b. In standard form, we have
$\frac{x'^2}{9/26}+\frac{y'^2}{9}=1$
which represents an ellipse.
c. We can graph the equation as shown in the figure.
