Answer
$$\eqalign{
& \left( {\text{a}} \right)\frac{{2 - \sqrt 2 }}{3} \cr
& \left( {\text{b}} \right)\frac{{2 - \sqrt 2 }}{3} \cr} $$
Work Step by Step
$$\eqalign{
& \int_0^1 {\frac{{{x^3}}}{{\sqrt {{x^2} + 1} }}} dx \cr
& \left( {\text{a}} \right) \cr
& \int {\frac{{{x^3}}}{{\sqrt {{x^2} + 1} }}dx = \int {\left( {{x^2}} \right)\left( {\frac{x}{{\sqrt {{x^2} + 1} }}} \right)} } dx \cr
& {\text{Let }}u = {x^2},{\text{ }}du = 2xdx \cr
& dv = \frac{x}{{\sqrt {{x^2} + 1} }}dx,{\text{ }}v = \sqrt {{x^2} + 1} \cr
& {\text{Using the integration by parts formula}} \cr
& \int {\frac{{{x^3}}}{{\sqrt {{x^2} + 1} }}dx} = {x^2}\sqrt {{x^2} + 1} - \int {\sqrt {{x^2} + 1} \left( {2x} \right)dx} \cr
& {\text{ }} = {x^2}\sqrt {{x^2} + 1} - \frac{{{{\left( {{x^2} + 1} \right)}^{3/2}}}}{{3/2}} + C \cr
& {\text{ }} = {x^2}\sqrt {{x^2} + 1} - \frac{{2{{\left( {{x^2} + 1} \right)}^{3/2}}}}{3} + C \cr
& {\text{Therefore,}} \cr
& \int_0^1 {\frac{{{x^3}}}{{\sqrt {{x^2} + 1} }}} dx = \left[ {{x^2}\sqrt {{x^2} + 1} - \frac{{2{{\left( {{x^2} + 1} \right)}^{3/2}}}}{3}} \right]_0^1 \cr
& = \left[ {{1^2}\sqrt {{1^2} + 1} - \frac{{2{{\left( {{1^2} + 1} \right)}^{3/2}}}}{3}} \right] - \left[ {{0^2}\sqrt {{0^2} + 1} - \frac{{2{{\left( {{0^2} + 1} \right)}^{3/2}}}}{3}} \right] \cr
& = \sqrt 2 - \frac{2}{3}\left( {2\sqrt 2 } \right) + \frac{2}{3} \cr
& = \frac{2}{3} - \frac{1}{3}\sqrt 2 \cr
& = \frac{{2 - \sqrt 2 }}{3} \cr
& \cr
& \left( {\text{b}} \right) \cr
& \int {\frac{{{x^3}}}{{\sqrt {{x^2} + 1} }}dx} \cr
& {\text{Let }}u = \sqrt {{x^2} + 1} ,{\text{ }}du = \frac{x}{{\sqrt {{x^2} + 1} }}dx \cr
& \int {\frac{{{x^3}}}{{\sqrt {{x^2} + 1} }}dx} = \int {{x^2}du} \cr
& u = \sqrt {{x^2} + 1} \Rightarrow {x^2} = {u^2} - 1 \cr
& \int {{x^2}du} = \int {\left( {{u^2} - 1} \right)du} \cr
& {\text{Integrating}} \cr
& {\text{ = }}\frac{1}{3}{u^3} - u + C \cr
& {\text{Back - substitute }}u = \sqrt {{x^2} + 1} \cr
& {\text{ = }}\frac{1}{3}{\left( {\sqrt {{x^2} + 1} } \right)^3} - \sqrt {{x^2} + 1} + C \cr
& \int_0^1 {\frac{{{x^3}}}{{\sqrt {{x^2} + 1} }}} dx = \left[ {\frac{1}{3}{{\left( {\sqrt {{x^2} + 1} } \right)}^3} - \sqrt {{x^2} + 1} } \right]_0^1 \cr
& = \left[ {\frac{1}{3}{{\left( {\sqrt {{1^2} + 1} } \right)}^3} - \sqrt {{1^2} + 1} } \right] - \left[ {\frac{1}{3}{{\left( {\sqrt {{0^2} + 1} } \right)}^3} - \sqrt {{0^2} + 1} } \right] \cr
& = \frac{1}{3}\left( {2\sqrt 2 } \right) - \sqrt 2 - \frac{1}{3} + 1 \cr
& = - \frac{1}{3}\sqrt 2 + \frac{2}{3} \cr
& = \frac{{2 - \sqrt 2 }}{3} \cr} $$