Answer
${I_0} = \frac{1}{{240}}{a^2}{b^2}\left( {5{a^2} + 4{b^2}} \right)$
Work Step by Step
We have the domain ${\cal D}$, between the line $y = \frac{b}{a}x$ and the parabola $y = \frac{b}{{{a^2}}}{x^2}$, where $a,b > 0$. The mass density is $\delta \left( {x,y} \right) = xy$.
We find the intersection of the line $y = \frac{b}{a}x$ and the parabola $y = \frac{b}{{{a^2}}}{x^2}$:
$y = \frac{b}{a}x = \frac{b}{{{a^2}}}{x^2}$
Multiplying both sides by $\frac{{{a^2}}}{b}$ gives
$ax = {x^2}$
${x^2} - ax = 0$
$x\left( {x - a} \right) = 0$
So, the intersections occur at $x=0$ and $x=a$.
We describe ${\cal D}$ as a vertically simple domain such that the description is given by:
${\cal D} = \left\{ {\left( {x,y} \right)|0 \le x \le a,\frac{b}{{{a^2}}}{x^2} \le y \le \frac{b}{a}x} \right\}$
1. Evaluate the moment of inertia relative to the $x$-axis:
${I_x} = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} {y^2}\delta \left( {x,y} \right){\rm{d}}A$
$ = \mathop \smallint \limits_{x = 0}^a \mathop \smallint \limits_{y = b{x^2}/{a^2}}^{bx/a} x{y^3}{\rm{d}}y{\rm{d}}x$
$ = \frac{1}{4}\mathop \smallint \limits_{x = 0}^a x\left( {{y^4}|_{b{x^2}/{a^2}}^{bx/a}} \right){\rm{d}}x$
$ = \frac{1}{4}\mathop \smallint \limits_{x = 0}^a \left( {\frac{{{b^4}}}{{{a^4}}}{x^5} - \frac{{{b^4}}}{{{a^8}}}{x^9}} \right){\rm{d}}x$
$ = \frac{{{b^4}}}{{4{a^4}}}\left( {\left( {\frac{1}{6}{x^6} - \frac{1}{{10{a^4}}}{x^{10}}} \right)|_0^a} \right)$
$ = \frac{{{b^4}}}{{4{a^4}}}\left( {\frac{{{a^6}}}{6} - \frac{{{a^6}}}{{10}}} \right) = \frac{{{a^2}{b^4}}}{{60}}$
2. Evaluate the moment of inertia relative to the $y$-axis:
${I_y} = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} {x^2}\delta \left( {x,y} \right){\rm{d}}A$
$ = \mathop \smallint \limits_{x = 0}^a \mathop \smallint \limits_{y = b{x^2}/{a^2}}^{bx/a} {x^3}y{\rm{d}}y{\rm{d}}x$
$ = \frac{1}{2}\mathop \smallint \limits_{x = 0}^a {x^3}\left( {{y^2}|_{b{x^2}/{a^2}}^{bx/a}} \right){\rm{d}}x$
$ = \frac{1}{2}\mathop \smallint \limits_{x = 0}^a \left( {\frac{{{b^2}}}{{{a^2}}}{x^5} - \frac{{{b^2}}}{{{a^4}}}{x^7}} \right){\rm{d}}x$
$ = \frac{{{b^2}}}{{2{a^2}}}\left( {\left( {\frac{1}{6}{x^6} - \frac{1}{{8{a^2}}}{x^8}} \right)|_0^a} \right)$
$ = \frac{{{b^2}}}{{2{a^2}}}\left( {\frac{{{a^6}}}{6} - \frac{{{a^6}}}{8}} \right) = \frac{{{a^4}{b^2}}}{{48}}$
Since, ${I_0} = {I_x} + {I_y}$, so
${I_0} = {I_x} + {I_y} = \frac{{{a^2}{b^4}}}{{60}} + \frac{{{a^4}{b^2}}}{{48}} = \frac{1}{{240}}{a^2}{b^2}\left( {5{a^2} + 4{b^2}} \right)$
