Answer
$$\frac{1}{4}{\sin ^{ - 1}}\sqrt x - \frac{1}{4}\sqrt x \sqrt {1 - x} \left( {1 - 2x} \right) + C$$
Work Step by Step
$$\eqalign{
& \int {\sqrt x \sqrt {1 - x} } dx \cr
& {\text{Taking the hint }}x = {u^2} \cr
& x = {u^2},\,\,\,\,\,\,\,dx = 2udu \cr
& {\text{Then}}{\text{,}} \cr
& \int {\sqrt x \sqrt {1 - x} } dx = \int {\sqrt {{u^2}} \sqrt {1 - {u^2}} } \left( {2udu} \right) \cr
& {\text{simplifying}} \cr
& = \int {u\sqrt {1 - {u^2}} } \left( {2udu} \right) \cr
& = 2\int {{u^2}\sqrt {1 - {u^2}} } du \cr
& {\text{Integrate by trigonometric substitution}} \cr
& {\text{We set}}{\text{, }}u = \sin \theta ,{\text{ }}du = \cos \theta d\theta \cr
& 2\int {{u^2}\sqrt {1 - {u^2}} } du = 2\int {{{\left( {\sin \theta } \right)}^2}\sqrt {1 - {{\left( {\sin \theta } \right)}^2}} } \left( {\cos \theta } \right)d\theta \cr
& = 2\int {{{\sin }^2}\theta \sqrt {1 - {{\sin }^2}\theta } } \cos \theta d\theta \cr
& {\text{use the fundamental identity }}{\cos ^2}\theta + {\sin ^2}\theta = 1 \cr
& = 2\int {{{\sin }^2}\theta \sqrt {{{\cos }^2}\theta } } \cos \theta d\theta \cr
& = 2\int {{{\sin }^2}\theta } {\cos ^2}\theta d\theta \cr
& = \frac{1}{2}{\int {\left( {2\sin \theta \cos \theta } \right)} ^2}d\theta \cr
& {\text{Use the identity }}\sin 2\theta = 2\sin \theta \cos \theta \cr
& = \frac{1}{2}\int {{{\sin }^2}} 2\theta d\theta \cr
& = \frac{1}{2}\int {\frac{{1 - \cos 4\theta }}{2}} d\theta \cr
& = \frac{1}{4}\int {\left( {1 - \cos 4\theta } \right)} d\theta \cr
& {\text{Integrating, we get}} \cr
& = \frac{1}{4}\left( {\theta - \frac{1}{4}\sin 4\theta } \right) + C \cr
& = \frac{1}{4}\theta - \frac{1}{{16}}\sin 4\theta + C \cr
& = \frac{1}{4}\theta - \frac{1}{{16}}\left( {\sin 2\left( {2\theta } \right)} \right) + C \cr
& {\text{use the identity }}\sin 2x = 2\sin x\cos x \cr
& = \frac{1}{4}\theta - \frac{1}{{16}}\left( {2\sin 2\theta \cos 2\theta } \right) + C \cr
& = \frac{1}{4}\theta - \frac{1}{{16}}\left( {2\left( {2\sin \theta \cos \theta } \right)\cos 2\theta } \right) + C \cr
& = \frac{1}{4}\theta - \frac{4}{{16}}\sin \theta \cos \theta \cos 2\theta + C \cr
& {\text{use the identity }}\cos 2x = 1 - 2{\sin ^2}x \cr
& = \frac{1}{4}\theta - \frac{1}{4}\sin \theta \cos \theta \left( {1 - 2{{\sin }^2}\theta } \right) + C \cr
& \cr
& {\text{we have that }}u = \sin \theta {\text{; then }}\theta = {\sin ^{ - 1}}\left( u \right){\text{ and cos}}\theta = \sqrt {1 - {u^2}} \cr
& {\text{write in terms of }}u \cr
& = \frac{1}{4}{\sin ^{ - 1}}u - \frac{1}{4}u\sqrt {1 - {u^2}} \left( {1 - 2{{\left( u \right)}^2}} \right) + C \cr
& \cr
& {\text{write in terms of }}x{\text{; replace }}u = \sqrt x \cr
& = \frac{1}{4}{\sin ^{ - 1}}\sqrt x - \frac{1}{4}\sqrt x \sqrt {1 - {{\left( {\sqrt x } \right)}^2}} \left( {1 - 2{{\left( {\sqrt x } \right)}^2}} \right) + C \cr
& = \frac{1}{4}{\sin ^{ - 1}}\sqrt x - \frac{1}{4}\sqrt x \sqrt {1 - x} \left( {1 - 2x} \right) + C \cr} $$