Answer
$$ \frac{1}{4}x^4\ln^2 x-\frac{1}{2}\left(\frac{1}{4}x^4\ln x-\frac{1}{16}x^4\right)+C$$
Work Step by Step
Given $$\int x^{3} \ln ^{2} x d x$$
Let
\begin{align*}
u&=\ln ^2x\ \ \ \ \ \ \ \ \ \ dv=x^3dx\\
du&= \frac{2\ln x }{x}dx\ \ \ \ \ \ \ v = \frac{1}{4}x^4
\end{align*}
Then
\begin{align*}
\int x^{3} \ln^2 x d x&= \frac{1}{4}x^4\ln^2 x-\frac{1}{2}\int x^3\ln xdx
\end{align*}
Let
\begin{align*}
u&=\ln x\ \ \ \ \ \ \ \ \ \ dv=x^3dx\\
du&= \frac{1 }{x}dx\ \ \ \ \ \ \ v = \frac{1}{4}x^4
\end{align*}
Then
\begin{align*}
\int x^3\ln xdx &= \frac{1}{4}x^4\ln x-\frac{1}{4}\int x^3dx\\
&= \frac{1}{4}x^4\ln x-\frac{1}{16}x^4
\end{align*}
Hence
\begin{align*}
\int x^{3} \ln^2 x d x&= \frac{1}{4}x^4\ln^2 x-\frac{1}{2}\left(\frac{1}{4}x^4\ln x-\frac{1}{16}x^4\right)+C
\end{align*}
