Answer
$$\int^{1/\sqrt2}_02x\sin^{-1}(x^2)dx=\frac{\pi}{12}+\frac{\sqrt3}{2}-1$$
Work Step by Step
$$A=\int^{1/\sqrt2}_02x\sin^{-1}(x^2)dx$$
We set $a=x^2$, which means $$da=2xdx$$
Also, for $x=1/\sqrt2$, $a=x^2=1/2$ and for $x=0$, $a=x^2=0$
Therefore, $$A=\int^{1/2}_0\sin^{-1}ada$$
Now set $u=\sin^{-1}a$ and $dv=da$
We have $du=\frac{1}{\sqrt{1-a^2}}da$ and $v=a$
Applying the formula $\int^b_a udv=uv]^b_a-\int^b_a vdu$, we have
$$A=a\sin^{-1}a\Big]^{1/2}_0-\int^{1/2}_0\frac{a}{\sqrt{1-a^2}}da$$ $$A=\frac{1}{2}\sin^{-1}\frac{1}{2}-\int^{1/2}_0\frac{a}{\sqrt{1-a^2}}da$$ $$A=\frac{1}{2}\times\frac{\pi}{6}-\int^{1/2}_0\frac{a}{\sqrt{1-a^2}}da$$ $$A=\frac{\pi}{12}-\int^{1/2}_0\frac{a}{\sqrt{1-a^2}}da$$
Take $n=\sqrt{1-a^2}$, then $$dn=\frac{-2a}{2\sqrt{1-a^2}}da=-\frac{a}{\sqrt{1-a^2}}da$$ $$\frac{a}{\sqrt{1-a^2}}da=-dn$$
Also, for $a=1/2$, $n=\sqrt3/2$ and for $a=0$, $n=1$.
Therefore, $$A=\frac{\pi}{12}-\int^{\sqrt3/2}_1-dn$$ $$A=\frac{\pi}{12}+n\Big]^{\sqrt3/2}_1$$ $$A=\frac{\pi}{12}+\frac{\sqrt3}{2}-1$$
