Answer
$\frac{x'^2}{3}-\frac{y'^2}{2}=1$, asymptotes $y' =\pm\frac{\sqrt 6}{3}x'$
Work Step by Step
Step 1. Given
$x^{2}+4xy-2y^{2}-6=0$
we have $A=1, B=4, C=-2$
and the rotation angle is
$cot(2\theta)=\frac{A-C}{B}=\frac{1+2}{4}=\frac{3}{4}$
Step 2. We have the transformations (see also exercise 15):
$x=\frac{\sqrt 5}{5}(2x'-y')$ and $y=\frac{\sqrt 5}{5}(x'+2y')$
Step 3. The equation becomes
$x^{2}+4xy-2y^{2}-6=(\frac{\sqrt 5}{5}(2x'-y'))^{2}+4(\frac{\sqrt 5}{5}(2x'-y'))(\frac{\sqrt 5}{5}(x'+2y'))-2(\frac{\sqrt 5}{5}(x'+2y'))^{2}-6=2x'^2-3y'^2-6=0$
or, $2x'^2-3y'^2=6$
Step 4. In standard form, we have
$\frac{x'^2}{3}-\frac{y'^2}{2}=1$
which represents a hyperbola.
Step 5. We have $a=\sqrt 3, b=\sqrt 2$ and the asymptotes in the $x'-y'$ system as $y'=\pm\frac{b}{a}x'=\pm\frac{\sqrt 6}{3}x'$
