Answer
$$ - \frac{{{x^3}}}{{3{{\left( {{x^2} - 1} \right)}^{3/2}}}} + C $$
Work Step by Step
$$\eqalign{
& \int {\frac{{{x^2}dx}}{{{{\left( {{x^2} - 1} \right)}^{5/2}}}}} \cr
& {\text{Use trigonometric substitution:}} \cr
& {\text{ }}x = \sec \theta,{\text{ }}dx = \sec \theta \tan \theta d\theta \cr
& {\text{With these substitutions}}{\text{, we have}} \cr
& \int {\frac{{{x^2}dx}}{{{{\left( {{x^2} - 1} \right)}^{5/2}}}}} = \int {\frac{{{{\sec }^2}\theta \left( {\sec \theta \tan \theta d\theta } \right)}}{{{{\left( {{{\sec }^2}\theta - 1} \right)}^{5/2}}}}} \cr
& = \int {\frac{{{{\sec }^3}\theta \tan \theta }}{{{{\left( {{{\sec }^2}\theta - 1} \right)}^{5/2}}}}} d\theta \cr
& {\text{Use the fundamental identity }}{\sec ^2}\theta - 1 = {\tan ^2}\theta \cr
& = \int {\frac{{{{\sec }^3}\theta \tan \theta }}{{{{\left( {{{\tan }^2}\theta } \right)}^{5/2}}}}} d\theta \cr
& = \int {\frac{{{{\sec }^3}\theta \tan \theta }}{{{{\tan }^5}\theta }}} d\theta \cr
& {\text{simplifying, we get:}} \cr
& = \int {\frac{{{{\sec }^3}\theta }}{{{{\tan }^4}\theta }}} d\theta \cr
& = \int {\left( {\frac{1}{{{{\cos }^3}\theta }}} \right)\left( {\frac{{{{\cos }^4}\theta }}{{{{\sin }^4}\theta }}} \right)} d\theta \cr
& = \int {\frac{{\cos \theta }}{{{{\sin }^4}\theta }}} d\theta \cr
& = \int {{{\left( {\sin \theta } \right)}^{ - 4}}\cos \theta } d\theta \cr
& {\text{integrate}} \cr
& = \frac{{{{\left( {\sin \theta } \right)}^{ - 3}}}}{{ - 3}} + C \cr
& = - \frac{1}{{3{{\sin }^3}\theta }} + C \cr
& {\text{write in terms of }}x,{\text{ }}\sin \theta = \frac{{\sqrt {{x^2} - 1} }}{x} \cr
& = - \frac{1}{3}{\left( {\frac{x}{{\sqrt {{x^2} - 1} }}} \right)^3} + C \cr
& = - \frac{{{x^3}}}{{3{{\left( {{x^2} - 1} \right)}^{3/2}}}} + C \cr} $$
