Answer
a) $\rho=5\epsilon_okr^{2}$
b)$Q_{enc}=4\pi\epsilon_okR^5$
Work Step by Step
a) $\rho=\epsilon_o\vec{\nabla}\cdot \vec{E}$
$\vec{E}=kr^3\hat{r}$
$\vec{\nabla}\cdot \vec{E}=\large\frac{1}{r^2}\frac{\partial}{\partial r}(\normalsize r^2E_r)+\large\frac{1}{r\sin{\theta}}\frac{\partial}{\partial \theta}(\normalsize\sin{\theta}E_\theta) +\large\frac{1}{r\sin{\theta}}\frac{\partial}{\partial \phi}(\normalsize E_\phi)$
$\because E_\theta=0$ , $E_\phi=0$ and $E_r=E\space(\equiv\left\lvert{\vec E}\right\rvert)$
$\therefore\vec{\nabla}\cdot \vec{E}=\large\frac{1}{r^2}\frac{\partial}{\partial r}(\normalsize r^2E)=\large\frac{1}{r^2}\frac{\partial}{\partial r}(\normalsize kr^5)=5kr^2$
$\Rightarrow\rho=5\epsilon_okr^{2}$
b)
i)First way
According to Gauss's law,
$Q_{enc}=\epsilon_o\oint\vec{E}\cdot{\vec da}$
${\vec da}=R^2\sin{\theta}d\theta d\phi\hat{r}$
$\Rightarrow Q_{enc}=kR^5\epsilon_o\int_{0}^{2\pi}d\phi\int_{0}^{\pi}\sin{\theta}d\theta=4\pi\epsilon_okR^5$
ii)Second way
$Q_{enc}=\int\rho d\tau=\int_{0}^{2\pi}\int_{0}^{\pi}\int_{0}^{R}(5\epsilon_okr^{2})(r^2\sin{\theta}drd\theta d\phi)$
$\hspace{0.9cm}=\int_{0}^{2\pi}d\phi\int_{0}^{\pi}\sin{\theta}d\theta\int_{0}^{R}(5\epsilon_okr^{4})dr=4\pi\epsilon_okR^5$
