Answer
$$\frac{1}{3} x^{3}(\ln x)^{2}-\frac{2}{9}\left[x^{3} \ln x-\frac{x^{3}}{3}\right]+C$$
Work Step by Step
Given $$ \int x^{2}(\ln x)^{2} d x$$Let
\begin{align*}
u&= (\ln x)^{2} \ \ \ \ \ \ \ \ \ dv=x^{2}dx\\
du&= 2(\ln x) \frac{1}{x}dx \ \ \ \ \ \ \ \ \ dv=\frac{1}{3}x^{3}
\end{align*}
Then
\begin{align*}
\int x^{2}(\ln x)^{2} d x&= \frac{1}{3}x^{3} (\ln x)^{2}-\frac{2}{3}\int x^2\ln xdx \end{align*}
Let
\begin{align*}
u&=\ln x \ \ \ \ \ dv =x^2dx\\
du&=\frac{1}{x}dx \ \ \ \ \ v =\frac{1}{3}x^3
\end{align*}
Then
\begin{align*}
\int x^2\ln xdx&=\frac{1}{3}x^3\ln x- \frac{1}{3}\int x^2 dx\\
& =\frac{1}{3}x^3\ln x- \frac{1}{9} x^3
\end{align*}
Hence
$$ \int x^{2}(\ln x)^{2} d x=\frac{1}{3} x^{3}(\ln x)^{2}-\frac{2}{9}\left[x^{3} \ln x-\frac{x^{3}}{3}\right]+C$$
