Answer
$x= \frac{\sqrt 2}{2}(x'-y')$ and $y= \frac{\sqrt 2}{2}(x'+y')$
Work Step by Step
Step 1. From the given equation, we have $A=5, B=-8, C=5$. Thus we have $cot2\theta=\frac{A-C}{B}=0$, which gives $2\theta=\frac{\pi}{2}$ and the rotation angle $\theta=\frac{\pi}{4}$
Step 2. Using the transformation formula $x=x'cos\theta-y'sin\theta$ and $y=x'sin\theta+y'cos\theta$, we have
$x=x'cos45^\circ-y'sin45^\circ=\frac{\sqrt 2}{2}(x'-y')$
$y=x'sin45^\circ+y'cos45^\circ=\frac{\sqrt 2}{2}(x'+y')$
