Answer
The correct answer is:
(c) inside sphere A somewhere on the line joining the centers.
Work Step by Step
The center of mass of sphere A is at the center of sphere A, and the center of mass of sphere B is at the center of sphere B. Let $d$ be the distance between the two centers.
Let $5M$ be the mass of sphere A and let $M$ be the mass of sphere B. We can find the position of the center of mass from the center of sphere A along the straight line joining the centers.
$x_{cm} = \frac{(5M)(0)+(M)(d)}{5M+M} = \frac{d}{6}$
Measured from sphere A's center, the center of mass of the system is $\frac{1}{6}$ of the distance along the straight line joining the two centers.
The correct answer is:
(c) inside sphere A somewhere on the line joining the centers.