Answer
\begin{aligned}
\int x^2 \ln(x^2+1)dx&=\frac{x^3}{3}\ln(x^2+1)-\frac{2}{3}\left( \frac{1}{3}x^3 -x + \tan^{-1}(x)+c\right)
\end{aligned}
Work Step by Step
Given
$$ \int x^2\ln(x^2+1)dx $$
Use integration by parts , let
\begin{aligned}
u&= \ln(x^2+1)\ \ \ \ \ \ \ &dv&= x^2dx\\
du&= \frac{2xdx}{x^2+1}\ \ \ \ \ \ \ & v&= \frac{x^3}{3}
\end{aligned}
Then
\begin{aligned}
\int \ln(x^2+1)dx&=\frac{x^3}{3}\ln(x^2+1)-\frac{2}{3}\int \frac{x^4dx}{x^2+1}
\end{aligned}
To evaluate $\int \dfrac{x^4dx}{x^2+1}$, let
$$x=\tan \theta \ \ \ \ \ dx=\sec^2 \theta d\theta $$
Then
\begin{aligned}
\int\dfrac{x^4dx}{x^2+1}&=\int \dfrac{\tan^4\theta \sec^2\theta d\theta }{\tan^2\theta +1}\\
&=\int \tan^4\theta d\theta \\
&=\int( \sec^2 \theta-1) \tan^2 \theta d\theta \\
&=\int \sec^2 \theta\tan^2\theta d\theta-\int \tan^2 \theta d\theta \\
&=\int \sec^2 \theta\tan^2\theta d\theta-\int (\sec^2 \theta-1) d\theta \\
&=\frac{1}{3}\tan^3 \theta -\tan \theta + \theta +c\\
&= \frac{1}{3}x^3 -x + \tan^{-1}(x)+c
\end{aligned}
Hence
\begin{aligned}
\int x^2 \ln(x^2+1)dx&=\frac{x^3}{3}\ln(x^2+1)-\frac{2}{3}\left( \frac{1}{3}x^3 -x + \tan^{-1}(x)+c\right)
\end{aligned}
