Answer
$$\frac{2}{{45}}{\left( {{x^3} + 1} \right)^{3/2}}\left( {3{x^3} - 8} \right) + C$$
Work Step by Step
$$\eqalign{
& \int {{x^5}\sqrt {{x^3} + 1} } dx \cr
& {\text{write as}} \cr
& = \int {{x^3}\sqrt {{x^3} + 1} \left( {{x^2}dx} \right)} \cr
& {\text{Taking the hint on the book making a substitution of the form }}u = {\left( {x + a} \right)^{1/n}} \cr
& {\text{Then}}{\text{,}} \cr
& {\text{Take }}\,\,\,u = {\left( {{x^3} + 1} \right)^{1/2}} \to \,\,\,\,\,\,\,\,{u^2} = {x^3} + 1,\,\,\,\,\,\,2udu = 3{x^2}dx \cr
& \cr
& {\text{write the integrand in terms of }}u \cr
& = \int {{x^3}\sqrt {{x^3} + 1} \left( {{x^2}dx} \right)} = \int {\left( {{u^2} - 1} \right)u\left( {{x^2}} \right)\left( {\frac{{2udu}}{{3{x^2}}}} \right)} \cr
& = \frac{2}{3}\int {{u^2}\left( {{u^2} - 1} \right)du} \cr
& = \frac{2}{3}\int {\left( {{u^4} - {u^2}} \right)} du \cr
& \cr
& {\text{integrating}} \cr
& = \frac{2}{3}\left( {\frac{{{u^5}}}{5} - \frac{{{u^3}}}{3}} \right) + C \cr
& = \frac{2}{{45}}\left( {3{u^5} - 5{u^3}} \right) + C \cr
& = \frac{2}{{45}}{u^3}\left( {3{u^2} - 5} \right) + C \cr
& {\text{write the integrand in terms of }}x;{\text{ replace }}u = {\left( {{x^3} + 1} \right)^{1/2}} \cr
& = \frac{2}{{45}}{\left( {{{\left( {{x^3} + 1} \right)}^{1/2}}} \right)^3}\left( {3{{\left( {{{\left( {{x^3} + 1} \right)}^{1/2}}} \right)}^2} - 5} \right) + C \cr
& = \frac{2}{{45}}{\left( {{x^3} + 1} \right)^{3/2}}\left( {3{x^3} - 3 - 5} \right) + C \cr
& = \frac{2}{{45}}{\left( {{x^3} + 1} \right)^{3/2}}\left( {3{x^3} - 8} \right) + C \cr} $$
