Answer
a. $x'^2-9y'^2+36=0$
b. $\frac{y'^2}{4}-\frac{x'^2}{36}=1$
c. see graph
Work Step by Step
a. Step 1. Given $3xy-4y^{2}+18=0$, we have $A=0, B=3, C=-4$, and the rotation angle
$cot(2\theta)=\frac{A-C}{B}=\frac{0+4}{3}=\frac{4}{3}$,
Step 2. We have the transformations (see also exercise 18):
$x=\frac{\sqrt 10}{10}(3x'-y')$ and $y=\frac{\sqrt 10}{10}(x'+3y')$
Step 3. The equation becomes
$3xy-4y^{2}+18=3(\frac{\sqrt 10}{10}(3x'-y'))(\frac{\sqrt 10}{10}(x'+3y'))-4(\frac{\sqrt 10}{10}(x'+3y'))^{2}+18=\frac{1}{2}x'^2-\frac{9}{2}y'^2+18=0$
or, $x'^2-9y'^2+36=0$
b. In standard form, we have
$\frac{y'^2}{4}-\frac{x'^2}{36}=1$
which represents a hyperbola.
c. We can graph the equation as shown in the figure.
