Answer
$\frac{2}{5}(x^3+1)^{5/2}-\frac{2}{3}(x^3+1)^{3/2}+C$
Work Step by Step
Let u=$x^3+1=>du=3x^2dx$
and $x^3=u-1$
so $\int 3x^5 \sqrt{x^3+1}dx$
=$\int (u-1)\sqrt{u}$
=$\int (u^{3/2}-u^{1/2})du$
=$\frac{2}{5}u^{5/2}-\frac{2}{3}u^{3/2}+C$
=$\frac{2}{5}(x^3+1)^{5/2}-\frac{2}{3}(x^3+1)^{3/2}+C$
