Answer
$\int\sqrt{x}\sin^2(x^{3/2} -1)dx = \frac{1}{3}(x^{3/2} -1 - \frac{1}{2}\sin(2(x^{3/2} - 1))) + C$
Work Step by Step
$\int\sqrt{x}\sin^2(x^{3/2} -1)dx\space$ and $\space u = x^{3/2} - 1$
$du = \frac{3}{2}\sqrt{x}\cdot dx$
Doing the substitution $\space u = x^{3/2} - 1$
$\int\sin^2(u)\frac{2}{3}du\space\Rightarrow\space\frac{2}{3}\int\sin^2(u)du\space\space\space(\mathsf{a})$
Remember that $\sin^2(\theta) = \frac{1 - \cos(2\theta)}{2}$
$\int\frac{1 - \cos(2u)}{2}du\space\Rightarrow\space\frac{1}{2}\int(1-\cos(2u))du\space\space\space(\mathsf{b})$
Combining the two expressions ($\mathsf{a}$ and $\mathsf{b}$) :
$\frac{1}{3}\int (1 - cos(2u))du$
Applying the integrative rules:
$\frac{1}{3}\int 1du - \frac{1}{3}\int\cos(2u)du$
Calculating the first integral:
$\frac{1}{3}\int 1du = \frac{1}{3}u\space\space\space\mathsf{A}$
Calculating the second integral:
$-\frac{1}{3}\int\cos(2u)du$
First we need to use the substitution $h = 2u$ and $dh = 2du$
$-\frac{1}{3}\int\cos(h)\frac{dh}{2}\space\Rightarrow\space-\frac{1}{6}\int\cos(h)dh$
$-\frac{1}{6}\int\cos(h)dh = -\frac{1}{6}\sin(h)$
Backing to $u$
$-\frac{1}{6}\sin(2u)\space\space\space\mathsf{B}$
Combining the two expressions ($\mathsf{A}$ and $\mathsf{B}$) :
$\frac{1}{3}u - \frac{1}{6}\sin(2u)\space\Rightarrow\space\frac{1}{3}(u - \frac{1}{2}\sin(2u))$
Backing to x
$\int\sqrt{x}\sin^2(x^{3/2} -1)dx = \frac{1}{3}(x^{3/2} -1 - \frac{1}{2}\sin(2(x^{3/2} - 1))) + C$