Answer
$\frac{3}{8}\sqrt 2+\frac{1}{8}\ln({\sqrt {2}-1})$
Work Step by Step
$\int{cot^{2}x(csc^{3}x})dx$ = $\int{(csc^{2}x-1)csc^{3}x}dx$ = $\int{csc^{5}x}dx$ - $\int{csc^{3}x}dx$
use the reduction formula for csc^{m}x
$\int{cot^{2}x(csc^{3}x})dx$
= $[-\frac{1}{4}cotx(csc^{3}x)+\frac{3}{4}\int{csc^{3}x}dx]-\int{csc^{3}x}dx$
= $[-\frac{1}{4}cotx(csc^{3}x)-\frac{1}{4}\int{csc^{3}x}dx]$
= $-\frac{1}{4}cotx(csc^{3}x)-\frac{1}{4}(-\frac{1}{2}cotx(cscx)+\frac{1}{2}\int{cscx}dx)$
= $-\frac{1}{4}cotx(csc^{3}x)+\frac{1}{8}cotx(cscx)-\frac{1}{8}\ln|{cscx-cotx|}+C$
$\int_{\frac{\pi}{4}}^{\frac{\pi}{2}}{cot^{2}x(csc^{3}x})dx$
= $-\frac{1}{4}cotx(csc^{3}x)+\frac{1}{8}cotx(cscx)-\frac{1}{8}\ln|{cscx-cotx|}|_{\frac{\pi}{4}}^{\frac{\pi}{2}}$
= $-\frac{1}{4}cot{\frac{\pi}{2}}(csc^{3}{\frac{\pi}{2}})+\frac{1}{8}cot{\frac{\pi}{2}}(csc{\frac{\pi}{2}})-\frac{1}{8}\ln|{csc{\frac{\pi}{2}}-cot{\frac{\pi}{2}}|}$+$\frac{1}{4}cot{\frac{\pi}{4}}(csc^{3}{\frac{\pi}{4}})-\frac{1}{8}cot{\frac{\pi}{4}}(csc{\frac{\pi}{4}})+\frac{1}{8}\ln|{csc{\frac{\pi}{4}}-cot{\frac{\pi}{4}}|}$
= $\frac{3}{8}\sqrt 2+\frac{1}{8}\ln({\sqrt {2}-1})$
