Answer
a. $4x'^2-y'^2=1$
b. $\frac{x'^2}{1/4}-\frac{y'^2}{1}=1$
c. see graph
Work Step by Step
a. Step 1. Given
$11x^{2}+10\sqrt{3}xy+y^{2}-4=0$
we have
$A=11, B=10, C=1$
The rotation angle is
$cot(2\theta)=\frac{A-C}{B}=\frac{11-1}{10\sqrt 3}=\frac{\sqrt 3}{3}, 2\theta=\frac{\pi}{3}, \theta=\frac{\pi}{6}$
Step 2. We have the transformations (see also exercise 19):
$x=\frac{1}{2}(\sqrt 3x'-y')$ and $y=\frac{1}{2}(x'+\sqrt 3y')$
Step 3. The equation becomes
$11x^{2}+10\sqrt{3}xy+y^{2}-4=11(\frac{1}{2}(\sqrt 3x'-y'))^{2}+10\sqrt{3}(\frac{1}{2}(\sqrt 3x'-y'))(\frac{1}{2}(x'+\sqrt 3y'))+(\frac{1}{2}(x'+\sqrt 3y'))^{2}-4=16x'^2-4y'^2-4=0$
or
$4x'^2-y'^2=1$
b. In standard form, we have
$\frac{x'^2}{1/4}-\frac{y'^2}{1}=1$
which represents a hyperbola.
c. We can graph the equation as shown in the figure.
