Answer
$x= \frac{\sqrt {10}}{10}(3x'-y')$ and
$y= \frac{\sqrt {10}}{10}(x'+3y')$
Work Step by Step
Step 1. From the given equation, we have $A=0, B=3, C=-4$. Thus we have $cot2\theta=\frac{A-C}{B}=\frac{0+4}{3}=\frac{4}{3}$. Thus $2\theta$ is in Quadrant I and $cos2\theta=\frac{4}{\sqrt {3^2+4^2}}=\frac{4}{5}$ (form a right triangle with sides $3,4,5$ here).
Step 2. We can find values of $sin\theta$ and $\cos\theta$ as $sin\theta=\sqrt {\frac{1-cos2\theta}{2}}=\sqrt {\frac{1-(\frac{4}{5})}{2}}=\frac{\sqrt {10}}{10}$ and $cos\theta=\sqrt {1-sin^2\theta}=\frac{3\sqrt {10}}{10}$
Step 3. Using the transformation formula $x=x'cos\theta-y'sin\theta$ and $y=x'sin\theta+y'cos\theta$, we have
$x=x'(\frac{3\sqrt {10}}{10})-y'(\frac{\sqrt {10}}{10})=\frac{\sqrt {10}}{10}(3x'-y')$
$y=x'(\frac{\sqrt {10}}{10})+y'(\frac{3\sqrt {10}}{10})=\frac{\sqrt {10}}{10}(x'+3y')$
