Answer
Undefined.
Work Step by Step
We can find the cotangent of coterminal angle that lies between $0^{\circ}$ and $360^{\circ}$ for convenience.
$\cot 540^{\circ}= \cot(540^{\circ}-360^{\circ})=\cot 180^{\circ}$
The terminal side of $180^{\circ}$ angle passes through (-1,0). So $x=-1,y=0$ and $r=1$.
$\cot 180^{\circ}=\frac{x}{y}=\frac{-1}{0}$
As denominator is 0, $\cot 540^{\circ}=\cot 180^{\circ}$ is not defined.
