Answer
The center of mass: $\left( {{x_{CM}},{y_{CM}}} \right) = \left( {\frac{{24a}}{{35}},\frac{{3b}}{5}} \right)$.
Work Step by Step
We have the domain ${\cal D}$, between the line $y = \frac{b}{a}x$ and the parabola $y = \frac{b}{{{a^2}}}{x^2}$, where $a,b > 0$. The mass density is $\delta \left( {x,y} \right) = xy$.
We find the intersection of the line $y = \frac{b}{a}x$ and the parabola $y = \frac{b}{{{a^2}}}{x^2}$:
$y = \frac{b}{a}x = \frac{b}{{{a^2}}}{x^2}$
Multiplying both sides by $\frac{{{a^2}}}{b}$ gives
$ax = {x^2}$
${x^2} - ax = 0$
$x\left( {x - a} \right) = 0$
So, the intersections occur at $x=0$ and $x=a$.
We describe ${\cal D}$ as a vertically simple domain such that the description is given by:
${\cal D} = \left\{ {\left( {x,y} \right)|0 \le x \le a,\frac{b}{{{a^2}}}{x^2} \le y \le \frac{b}{a}x} \right\}$
1. Evaluate the total mass:
$M = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} \delta \left( {x,y} \right){\rm{d}}A = \mathop \smallint \limits_{x = 0}^a \mathop \smallint \limits_{y = b{x^2}/{a^2}}^{bx/a} xy{\rm{d}}y{\rm{d}}x$
$ = \frac{1}{2}\mathop \smallint \limits_{x = 0}^a x\left( {{y^2}|_{b{x^2}/{a^2}}^{bx/a}} \right){\rm{d}}x$
$ = \frac{1}{2}\mathop \smallint \limits_{x = 0}^a x\left( {\frac{{{b^2}}}{{{a^2}}}{x^2} - \frac{{{b^2}}}{{{a^4}}}{x^4}} \right){\rm{d}}x$
$ = \frac{{{b^2}}}{{2{a^2}}}\mathop \smallint \limits_{x = 0}^a \left( {{x^3} - \frac{1}{{{a^2}}}{x^5}} \right){\rm{d}}x$
$ = \frac{{{b^2}}}{{2{a^2}}}\left( {\left( {\frac{1}{4}{x^4} - \frac{1}{{6{a^2}}}{x^6}} \right)|_0^a} \right)$
$ = \frac{{{b^2}}}{{2{a^2}}}\left( {\frac{{{a^4}}}{4} - \frac{{{a^4}}}{6}} \right) = \frac{{{a^2}{b^2}}}{{24}}$
2. Evaluate the $x$-coordinate of the center of mass:
${x_{CM}} = \frac{{{M_y}}}{M} = \frac{1}{M}\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} x\delta \left( {x,y} \right){\rm{d}}A$
$ = \frac{{24}}{{{a^2}{b^2}}}\mathop \smallint \limits_{x = 0}^a \mathop \smallint \limits_{y = b{x^2}/{a^2}}^{bx/a} {x^2}y{\rm{d}}y{\rm{d}}x$
$ = \frac{{12}}{{{a^2}{b^2}}}\mathop \smallint \limits_{x = 0}^a {x^2}\left( {{y^2}|_{b{x^2}/{a^2}}^{bx/a}} \right){\rm{d}}x$
$ = \frac{{12}}{{{a^2}{b^2}}}\mathop \smallint \limits_{x = 0}^a {x^2}\left( {\frac{{{b^2}}}{{{a^2}}}{x^2} - \frac{{{b^2}}}{{{a^4}}}{x^4}} \right){\rm{d}}x$
$ = \frac{{12}}{{{a^4}}}\mathop \smallint \limits_{x = 0}^a \left( {{x^4} - \frac{1}{{{a^2}}}{x^6}} \right){\rm{d}}x$
$ = \frac{{12}}{{{a^4}}}\left( {\left( {\frac{1}{5}{x^5} - \frac{1}{{7{a^2}}}{x^7}} \right)|_0^a} \right)$
$ = \frac{{12}}{{{a^4}}}\left( {\frac{{{a^5}}}{5} - \frac{{{a^5}}}{7}} \right) = \frac{{24a}}{{35}}$
3. Evaluate the $y$-coordinate of the center of mass:
${y_{CM}} = \frac{{{M_x}}}{M} = \frac{1}{M}\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} y\delta \left( {x,y} \right){\rm{d}}A$
$ = \frac{{24}}{{{a^2}{b^2}}}\mathop \smallint \limits_{x = 0}^a \mathop \smallint \limits_{y = b{x^2}/{a^2}}^{bx/a} x{y^2}{\rm{d}}y{\rm{d}}x$
$ = \frac{8}{{{a^2}{b^2}}}\mathop \smallint \limits_{x = 0}^a x\left( {{y^3}|_{b{x^2}/{a^2}}^{bx/a}} \right){\rm{d}}x$
$ = \frac{8}{{{a^2}{b^2}}}\mathop \smallint \limits_{x = 0}^a x\left( {\frac{{{b^3}}}{{{a^3}}}{x^3} - \frac{{{b^3}}}{{{a^6}}}{x^6}} \right){\rm{d}}x$
$ = \frac{{8b}}{{{a^5}}}\mathop \smallint \limits_{x = 0}^a \left( {{x^4} - \frac{1}{{{a^3}}}{x^7}} \right){\rm{d}}x$
$ = \frac{{8b}}{{{a^5}}}\left( {\left( {\frac{1}{5}{x^5} - \frac{1}{{8{a^3}}}{x^8}} \right)|_0^a} \right)$
$ = \frac{{8b}}{{{a^5}}}\left( {\frac{{{a^5}}}{5} - \frac{{{a^5}}}{8}} \right) = \frac{{3b}}{5}$
Thus, the center of mass: $\left( {{x_{CM}},{y_{CM}}} \right) = \left( {\frac{{24a}}{{35}},\frac{{3b}}{5}} \right)$.
