Answer
The coordinates of the center of mass: $\left( {\frac{8}{{15}},\frac{{16}}{{15\pi }},\frac{{16}}{{15\pi }}} \right)$.
Work Step by Step
We use $\delta $ instead of $\rho $ to denote the mass density to avoid confusing with the radial coordinate $\rho $. So, we have $\delta \left( {x,y,z} \right) = x$.
Let ${\cal W}$ denote the ball ${x^2} + {y^2} + {z^2} = 1$ in the first octant. Using spherical coordinates, we can describe ${\cal W}$ as
${\cal W} = \left\{ {\left( {\rho ,\phi ,\theta } \right)|0 \le \rho \le 1,0 \le \phi \le \frac{\pi }{2},0 \le \theta \le \frac{\pi }{2}} \right\}$
In spherical coordinates:
$x = \rho \sin \phi \cos \theta $, ${\ \ \ }$ $y = \rho \sin \phi \sin \theta $, ${\ \ \ }$ $z = \rho \cos \phi $
$\delta \left( {\rho \sin \phi \cos \theta ,\rho \sin \phi \sin \theta ,\rho \cos \phi } \right) = \rho \sin \phi \cos \theta $
Evaluate the mass of ${\cal W}$:
$M = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal W}^{} \delta \left( {x,y,z} \right){\rm{d}}V$
$ = \mathop \smallint \limits_{\theta = 0}^{\pi /2} \mathop \smallint \limits_{\rho = 0}^1 \mathop \smallint \limits_{\phi = 0}^{\pi /2} {\rho ^3}{\sin ^2}\phi \cos \theta {\rm{d}}\phi {\rm{d}}\rho {\rm{d}}\theta $
$ = \left( {\mathop \smallint \limits_{\theta = 0}^{\pi /2} \cos \theta {\rm{d}}\theta } \right)\left( {\mathop \smallint \limits_{\rho = 0}^1 {\rho ^3}{\rm{d}}\rho } \right)\left( {\mathop \smallint \limits_{\phi = 0}^{\pi /2} {{\sin }^2}\phi {\rm{d}}\phi } \right)$
Using the Double-angle formulas in Section 1.4:
${\sin ^2}x = \frac{1}{2}\left( {1 - \cos 2x} \right)$
we get
$M = \left( {\mathop \smallint \limits_{\theta = 0}^{\pi /2} \cos \theta {\rm{d}}\theta } \right)\left( {\mathop \smallint \limits_{\rho = 0}^1 {\rho ^3}{\rm{d}}\rho } \right)\left( {\frac{1}{2}\mathop \smallint \limits_{\phi = 0}^{\pi /2} \left( {1 - \cos 2\phi } \right){\rm{d}}\phi } \right)$
$ = \frac{1}{8}\left( {\sin \theta |_0^{\pi /2}} \right)\left( {{\rho ^4}|_0^1} \right)\left( {\left( {\phi - \frac{1}{2}\sin 2\phi } \right)|_0^{\pi /2}} \right)$
$ = \frac{1}{8}\left( {\frac{\pi }{2}} \right) = \frac{\pi }{{16}}$
Evaluate ${x_{CM}}$:
${x_{CM}} = \frac{1}{M}\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal W}^{} x\delta \left( {x,y,z} \right){\rm{d}}V$
$ = \frac{{16}}{\pi }\mathop \smallint \limits_{\theta = 0}^{\pi /2} \mathop \smallint \limits_{\rho = 0}^1 \mathop \smallint \limits_{\phi = 0}^{\pi /2} {\rho ^4}{\sin ^3}\phi {\cos ^2}\theta {\rm{d}}\phi {\rm{d}}\rho {\rm{d}}\theta $
(1) ${\ \ \ \ }$ ${x_{CM}} = \frac{{16}}{\pi }\left( {\mathop \smallint \limits_{\theta = 0}^{\pi /2} {{\cos }^2}\theta {\rm{d}}\theta } \right)\left( {\mathop \smallint \limits_{\rho = 0}^1 {\rho ^4}{\rm{d}}\rho } \right)\left( {\mathop \smallint \limits_{\phi = 0}^{\pi /2} {{\sin }^3}\phi {\rm{d}}\phi } \right)$
1. Consider the first integral in the right-hand side:
$\mathop \smallint \limits_{\theta = 0}^{\pi /2} {\cos ^2}\theta {\rm{d}}\theta $
Using the Double-angle formulas in Section 1.4:
${\cos ^2}x = \frac{1}{2}\left( {1 + \cos 2x} \right)$
we get
$\mathop \smallint \limits_{\theta = 0}^{\pi /2} {\cos ^2}\theta {\rm{d}}\theta = \frac{1}{2}\mathop \smallint \limits_{\theta = 0}^{\pi /2} \left( {1 + \cos 2\theta } \right){\rm{d}}\theta $
(2) ${\ \ \ \ }$ $\mathop \smallint \limits_{\theta = 0}^{\pi /2} {\cos ^2}\theta {\rm{d}}\theta = \frac{1}{2}\left( {\theta + \frac{1}{2}\sin 2\theta } \right)|_0^{\pi /2} = \frac{\pi }{4}$
2. Consider the third integral in the right-hand side:
$\mathop \smallint \limits_{\phi = 0}^{\pi /2} {\sin ^3}\phi {\rm{d}}\phi $
Using Eq. (5) of the Table of Trigonometric Integrals in Section 8.2:
$\smallint {\sin ^n}x{\rm{d}}x = - \frac{{{{\sin }^{n - 1}}x\cos x}}{n} + \frac{{n - 1}}{n}\smallint {\sin ^{n - 2}}x{\rm{d}}x$
we get
$\mathop \smallint \limits_{\phi = 0}^{\pi /2} {\sin ^3}\phi {\rm{d}}\phi = - \frac{{{{\sin }^2}\phi \cos \phi }}{3}|_0^{\pi /2} + \frac{2}{3}\mathop \smallint \limits_{\phi = 0}^{\pi /2} \sin \phi {\rm{d}}\phi $
(3) ${\ \ \ \ }$ $\mathop \smallint \limits_{\phi = 0}^{\pi /2} {\sin ^3}\phi {\rm{d}}\phi = - \frac{2}{3}\left( {\cos \phi |_0^{\pi /2}} \right) = \frac{2}{3}$
Substituting the results in equations (2) and (3) back in equation (1) gives
${x_{CM}} = \frac{{16}}{\pi }\left( {\frac{\pi }{4}} \right)\left( {\frac{1}{5}{\rho ^5}|_0^1} \right)\left( {\frac{2}{3}} \right)$
$ = \frac{{16}}{\pi }\left( {\frac{\pi }{4}} \right)\left( {\frac{1}{5}} \right)\left( {\frac{2}{3}} \right) = \frac{8}{{15}}$
So, ${x_{CM}} = \frac{8}{{15}}$.
Evaluate ${y_{CM}}$:
${y_{CM}} = \frac{1}{M}\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal W}^{} y\delta \left( {x,y,z} \right){\rm{d}}V$
$ = \frac{{16}}{\pi }\mathop \smallint \limits_{\theta = 0}^{\pi /2} \mathop \smallint \limits_{\rho = 0}^1 \mathop \smallint \limits_{\phi = 0}^{\pi /2} {\rho ^4}{\sin ^3}\phi \cos \theta \sin \theta {\rm{d}}\phi {\rm{d}}\rho {\rm{d}}\theta $
${y_{CM}} = \frac{{16}}{\pi }\left( {\mathop \smallint \limits_{\theta = 0}^{\pi /2} \cos \theta \sin \theta {\rm{d}}\theta } \right)\left( {\mathop \smallint \limits_{\rho = 0}^1 {\rho ^4}{\rm{d}}\rho } \right)\left( {\mathop \smallint \limits_{\phi = 0}^{\pi /2} {{\sin }^3}\phi {\rm{d}}\phi } \right)$
Using equation (3): $\mathop \smallint \limits_{\phi = 0}^{\pi /2} {\sin ^3}\phi {\rm{d}}\phi = \frac{2}{3}$,
we get
${y_{CM}} = \frac{{16}}{\pi }\left( {\frac{1}{2}\mathop \smallint \limits_{\theta = 0}^{\pi /2} \sin 2\theta {\rm{d}}\theta } \right)\left( {\mathop \smallint \limits_{\rho = 0}^1 {\rho ^4}{\rm{d}}\rho } \right)\left( {\frac{2}{3}} \right)$
$ = \frac{{16}}{{3\pi }}\left( { - \frac{1}{2}\left( {\cos 2\theta |_0^{\pi /2}} \right)} \right)\left( {\frac{1}{5}{\rho ^5}|_0^1} \right)$
$ = \frac{{16}}{{3\pi }}\left( 1 \right)\left( {\frac{1}{5}} \right) = \frac{{16}}{{15\pi }}$
So, ${y_{CM}} = \frac{{16}}{{15\pi }}$.
Evaluate ${z_{CM}}$:
${z_{CM}} = \frac{1}{M}\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{\cal W}^{} z\delta \left( {x,y,z} \right){\rm{d}}V$
$ = \frac{{16}}{\pi }\mathop \smallint \limits_{\theta = 0}^{\pi /2} \mathop \smallint \limits_{\rho = 0}^1 \mathop \smallint \limits_{\phi = 0}^{\pi /2} {\rho ^4}{\sin ^2}\phi \cos \phi \cos \theta {\rm{d}}\phi {\rm{d}}\rho {\rm{d}}\theta $
$ = \frac{{16}}{\pi }\left( {\mathop \smallint \limits_{\theta = 0}^{\pi /2} \cos \theta {\rm{d}}\theta } \right)\left( {\mathop \smallint \limits_{\rho = 0}^1 {\rho ^4}{\rm{d}}\rho } \right)\left( {\mathop \smallint \limits_{\phi = 0}^{\pi /2} {{\sin }^2}\phi \cos \phi {\rm{d}}\phi } \right)$
$ = \frac{{16}}{\pi }\left( {\mathop \smallint \limits_{\theta = 0}^{\pi /2} \cos \theta {\rm{d}}\theta } \right)\left( {\mathop \smallint \limits_{\rho = 0}^1 {\rho ^4}{\rm{d}}\rho } \right)\left( {\mathop \smallint \limits_{\phi = 0}^{\pi /2} {{\sin }^2}\phi {\rm{d}}\left( {\sin \phi } \right)} \right)$
$ = \frac{{16}}{\pi }\left( {\sin \theta |_0^{\pi /2}} \right)\left( {\frac{1}{5}{\rho ^5}|_0^1} \right)\left( {\frac{1}{3}\left( {{{\sin }^3}\phi |_0^{\pi /2}} \right)} \right)$
$ = \frac{{16}}{\pi }\left( 1 \right)\left( {\frac{1}{5}} \right)\left( {\frac{1}{3}} \right) = \frac{{16}}{{15\pi }}$
So, the coordinates of the center of mass: $\left( {\frac{8}{{15}},\frac{{16}}{{15\pi }},\frac{{16}}{{15\pi }}} \right)$.
