Answer
a. $2x'^2-3y'^2=1$
b. $\frac{x'^2}{1/2}-\frac{y'^2}{1/3}=1$
c. see graph
Work Step by Step
a. Step 1. Given
$x^{2}+4xy-2y^{2}-1=0$
we have
$A=1, B=4, C=-2$
The rotation angle is
$cot(2\theta)=\frac{A-C}{B}=\frac{1+2}{4}=\frac{3}{4}$,
Step 2. We have the transformations (see also exercise 23): $x=\frac{\sqrt 5}{5}(2x'-y')$ and $y=\frac{\sqrt 5}{5}(x'+2y')$
Step 3. The equation becomes
$x^{2}+4xy-2y^{2}-1=(\frac{\sqrt 5}{5}(2x'-y'))^{2}+4(\frac{\sqrt 5}{5}(2x'-y'))(\frac{\sqrt 5}{5}(x'+2y'))-2(\frac{\sqrt 5}{5}(x'+2y'))^{2}-1=2x'^2-3y'^2-1=0$
or, $2x'^2-3y'^2=1$
b. In standard form, we have $\frac{x'^2}{1/2}-\frac{y'^2}{1/3}=1$, which represents a hyperbola.
c. We can graph the equation as shown in the figure.
