Answer
$$\int_{0}^{1} \int_{0}^{1} x y \sqrt{x^{2}+y^{2}} d y d x= \frac{(\sqrt 2)^5-2}{15} $$
Work Step by Step
Given $$\int_{0}^{1} \int_{0}^{1} x y \sqrt{x^{2}+y^{2}} d y d x$$So we have
\begin{aligned}I&=\int_{0}^{1} \int_{0}^{1} x y \sqrt{x^{2}+y^{2}} d y d x\\
&=\frac{1}{2}\int_{0}^{1} \int_{0}^{1} x(2 y) \left(x^{2}+y^{2}\right)^{\frac{1}{2}} d y d x\\
&=\frac{1}{2}\int_{0}^{1} x (\frac{2}{3}) \left(x^{2}+y^{2}\right)^{\frac{3}{2}}|_{0}^{1} \ \ \ d x\\
&=\frac{1}{3}\int_{0}^{1} x \left(x^{2}+y^{2}\right)^{\frac{3}{2}}|_{0}^{1} \ \ \ d x\\
&=\frac{1}{3}\int_{0}^{1} x \left(x^{2}+1\right)^{\frac{3}{2}} \ dx - \frac{1}{3}\int_{0}^{1} x \left(x^{2}+0\right)^{\frac{3}{2}} \ \ \ d x\\
&=\frac{1}{3}\int_{0}^{1} x \left(x^{2}+1\right)^{\frac{3}{2}} \ dx - \frac{1}{3}\int_{0}^{1} x^4 \ \ \ d x\\
&=\frac{1}{6}\int_{0}^{1} 2x \left(x^{2}+1\right)^{\frac{3}{2}} \ dx - \frac{1}{6}\int_{0}^{1} x^4 \ \ \ d x\\
&=\frac{1}{6} \frac{2}{5} \left(x^{2}+1\right)^{\frac{5}{2}}|_{0}^{1} - \frac{1}{15} x^5|_{0}^{1} \ \ \\
&=\frac{1}{15} \left(1+1\right)^{\frac{5}{2}} - \frac{1}{15} \left(0+1\right)^{\frac{5}{2}} - \frac{1}{15} \ \ \\
&= \frac{(\sqrt 2)^5}{15} - \frac{2}{15} \\
&= \frac{(\sqrt 2)^5-2}{15}
\end{aligned}
