Answer
$(x^2+y^2)^2= x^3 - 3xy^2$
Work Step by Step
We have
\begin{align*}
\cos 3\theta &=\cos(\theta+2\theta)\\
&=\cos\theta \cos(2\theta)-\sin\theta \sin(2\theta)\\
&=\cos\theta (\cos^2\theta-\sin^2\theta)-2\sin^2\theta \cos\theta\\
&= \cos^3\theta-3\sin^2\theta \cos\theta.
\end{align*}
Now, we get the following
\begin{align*}
r &=\cos 3\theta\\
&= \cos^3\theta-3\sin^2\theta \cos\theta\\
&=\frac{x^3}{r^3}-\frac{3xy^2}{r^3}.
\end{align*}
That is
$$r^4= x^3 - 3xy^2 \Longrightarrow (x^2+y^2)^2= x^3 - 3xy^2.$$