Answer
$$\int x(\ln x)^2dx=\frac{x^2}{4}\Big(2(\ln x)^2-2\ln x+1\Big)+C$$
Work Step by Step
$$A=\int x(\ln x)^2dx$$
Set $a=\ln x$, which means $x=e^a$
We then have $$da=\frac{dx}{x}$$ $$dx=xda=e^ada$$
Therefore, $$A=\int e^a\times a^2\times e^ada=\int a^2e^{2a}da$$
Set $u=a^2$ and $dv=e^{2a}da$
So $du=2ada$ and $v=\frac{e^{2a}}{2}$
Following the formula $\int udv=uv-\int vdu$, we have
$$A=\frac{a^2e^{2a}}{2}-\int \frac{e^{2a}}{2}\times2ada$$ $$A=\frac{a^2e^{2a}}{2}-\int ae^{2a}da$$
Set $u=a$ and $dv=e^{2a}da$
So $du=da$ and $v=\frac{e^{2a}}{2}$
Following the formula $\int udv=uv-\int vdu$, we have
$$A=\frac{a^2e^{2a}}{2}-\Big(\frac{ae^{2a}}{2}-\frac{1}{2}\int e^{2a}da\Big)$$ $$A=\frac{a^2e^{2a}}{2}-\frac{ae^{2a}}{2}+\frac{1}{2}\int e^{2a}da$$ $$A=\frac{a^2e^{2a}}{2}-\frac{ae^{2a}}{2}+\frac{e^{2a}}{4}+C$$ $$A=\frac{e^{2a}}{4}(2a^2-2a+1)+C$$ $$A=\frac{e^{2\ln x}}{4}\Big(2(\ln x)^2-2\ln x+1\Big)+C$$
We have $e^{2\ln x}=(e^{\ln x})^2=x^2$
$$A=\frac{x^2}{4}\Big(2(\ln x)^2-2\ln x+1\Big)+C$$