Answer
The coordinates of the centroid: $\left( {\bar x,\bar y} \right) = \left( {\frac{a}{2},\frac{{2b}}{5}} \right)$.
Work Step by Step
We have the domain ${\cal D}$, between the line $y = \frac{b}{a}x$ and the parabola $y = \frac{b}{{{a^2}}}{x^2}$, where $a,b > 0$. The mass density is $\delta \left( {x,y} \right) = 1$.
We find the intersection of the line $y = \frac{b}{a}x$ and the parabola $y = \frac{b}{{{a^2}}}{x^2}$:
$y = \frac{b}{a}x = \frac{b}{{{a^2}}}{x^2}$
Multiplying both sides by $\frac{{{a^2}}}{b}$ gives
$ax = {x^2}$
${x^2} - ax = 0$
$x\left( {x - a} \right) = 0$
So, the intersections occur at $x=0$ and $x=a$.
We describe ${\cal D}$ as a vertically simple domain such that the description is given by:
${\cal D} = \left\{ {\left( {x,y} \right)|0 \le x \le a,\frac{b}{{{a^2}}}{x^2} \le y \le \frac{b}{a}x} \right\}$
By definition, the centroid is given by
$\bar x = \frac{1}{A}\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} x{\rm{d}}A$, ${\ \ \ }$ $\bar y = \frac{1}{A}\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} y{\rm{d}}A$,
where $A$ is the area of ${\cal D}$.
Evaluate the area:
$A = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} {\rm{d}}A = \mathop \smallint \limits_{x = 0}^a \mathop \smallint \limits_{y = b{x^2}/{a^2}}^{bx/a} {\rm{d}}y{\rm{d}}x$
$ = \mathop \smallint \limits_{x = 0}^a \left( {y|_{b{x^2}/{a^2}}^{bx/a}} \right){\rm{d}}x$
$ = \mathop \smallint \limits_{x = 0}^a \left( {\frac{b}{a}x - \frac{b}{{{a^2}}}{x^2}} \right){\rm{d}}x$
$ = \frac{b}{a}\mathop \smallint \limits_{x = 0}^a \left( {x - \frac{1}{a}{x^2}} \right){\rm{d}}x$
$ = \frac{b}{a}\left( {\left( {\frac{1}{2}{x^2} - \frac{1}{{3a}}{x^3}} \right)|_0^a} \right) = \frac{b}{a}\left( {\frac{{{a^2}}}{2} - \frac{{{a^2}}}{3}} \right) = \frac{{ab}}{6}$
Evaluate the average of $x$-coordinate:
$\bar x = \frac{1}{A}\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} x{\rm{d}}A = \frac{6}{{ab}}\mathop \smallint \limits_{x = 0}^a \mathop \smallint \limits_{y = b{x^2}/{a^2}}^{bx/a} x{\rm{d}}y{\rm{d}}x$
$ = \frac{6}{{ab}}\mathop \smallint \limits_{x = 0}^a x\left( {y|_{b{x^2}/{a^2}}^{bx/a}} \right){\rm{d}}x$
$ = \frac{6}{{ab}}\mathop \smallint \limits_{x = 0}^a x\left( {\frac{b}{a}x - \frac{b}{{{a^2}}}{x^2}} \right){\rm{d}}x$
$ = \frac{6}{{{a^2}}}\mathop \smallint \limits_{x = 0}^a \left( {{x^2} - \frac{1}{a}{x^3}} \right){\rm{d}}x$
$ = \frac{6}{{{a^2}}}\left( {\left( {\frac{1}{3}{x^3} - \frac{1}{{4a}}{x^4}} \right)|_0^a} \right) = \frac{6}{{{a^2}}}\left( {\frac{{{a^3}}}{3} - \frac{{{a^3}}}{4}} \right) = \frac{a}{2}$
Evaluate the average of $y$-coordinate:
$\bar y = \frac{1}{A}\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal D}^{} y{\rm{d}}A = \frac{6}{{ab}}\mathop \smallint \limits_{x = 0}^a \mathop \smallint \limits_{y = b{x^2}/{a^2}}^{bx/a} y{\rm{d}}y{\rm{d}}x$
$ = \frac{3}{{ab}}\mathop \smallint \limits_{x = 0}^a \left( {{y^2}|_{b{x^2}/{a^2}}^{bx/a}} \right){\rm{d}}x$
$ = \frac{3}{{ab}}\mathop \smallint \limits_{x = 0}^a \left( {\frac{{{b^2}}}{{{a^2}}}{x^2} - \frac{{{b^2}}}{{{a^4}}}{x^4}} \right){\rm{d}}x$
$ = \frac{{3b}}{{{a^3}}}\left( {\left( {\frac{1}{3}{x^3} - \frac{1}{{5{a^2}}}{x^5}} \right)|_0^a} \right) = \frac{{3b}}{{{a^3}}}\left( {\frac{{{a^3}}}{3} - \frac{{{a^3}}}{5}} \right) = \frac{{2b}}{5}$
So, the coordinates of the centroid: $\left( {\bar x,\bar y} \right) = \left( {\frac{a}{2},\frac{{2b}}{5}} \right)$.