Answer
We use polar coordinates to show that $f\left( {x,y} \right)$ is continuous at all $\left( {x,y} \right)$ if $p>2$ but is discontinuous at $\left( {0,0} \right)$ if $p \le 2$.
Work Step by Step
We are given the function:
\[f\left( {x,y} \right) = \left\{ {\begin{array}{*{20}{c}}
{\frac{{{{\left( {xy} \right)}^P}}}{{{x^4} + {y^4}}}}&{\left( {x,y} \right) \ne \left( {0,0} \right)}\\
0&{\left( {x,y} \right) = \left( {0,0} \right)}
\end{array}} \right.\]
Notice that $\frac{{{{\left( {xy} \right)}^P}}}{{{x^4} + {y^4}}}$ is continuous for $\left( {x,y} \right) \ne \left( {0,0} \right)$ since both the denominator and the numerator are continuous.
Evaluate the limit $\mathop {\lim }\limits_{\left( {x,y} \right) \to \left( {0,0} \right)} \frac{{{{\left( {xy} \right)}^P}}}{{{x^4} + {y^4}}}$ in polar coordinates:
$\mathop {\lim }\limits_{\left( {x,y} \right) \to \left( {0,0} \right)} \frac{{{{\left( {xy} \right)}^P}}}{{{x^4} + {y^4}}} = \mathop {\lim }\limits_{r \to 0} \frac{{{{\left( {{r^2}\cos \theta \sin \theta } \right)}^P}}}{{{r^4}{{\cos }^4}\theta + {r^4}{{\sin }^4}\theta }}$
$ = \mathop {\lim }\limits_{r \to 0} \frac{{{r^{2p}}}}{{{r^4}}}\frac{{{{\cos }^P}\theta {{\sin }^P}\theta }}{{{{\cos }^4}\theta + {{\sin }^4}\theta }}$
(1) ${\ \ \ \ }$ $\mathop {\lim }\limits_{\left( {x,y} \right) \to \left( {0,0} \right)} \frac{{{{\left( {xy} \right)}^P}}}{{{x^4} + {y^4}}} = \mathop {\lim }\limits_{r \to 0} {r^{2p - 4}}\frac{{{{\cos }^P}\theta {{\sin }^P}\theta }}{{{{\cos }^4}\theta + {{\sin }^4}\theta }}$
Case 1. $2p - 4 > 0$
In this case $p>2$. So,
$\mathop {\lim }\limits_{r \to 0} {r^{2p - 4}}\frac{{{{\cos }^P}\theta {{\sin }^P}\theta }}{{{{\cos }^4}\theta + {{\sin }^4}\theta }} = \frac{{{{\cos }^P}\theta {{\sin }^P}\theta }}{{{{\cos }^4}\theta + {{\sin }^4}\theta }}\mathop {\lim }\limits_{r \to 0} {r^{2p - 4}} = 0$
Thus, the limit (1) yields $\mathop {\lim }\limits_{\left( {x,y} \right) \to \left( {0,0} \right)} \frac{{{{\left( {xy} \right)}^P}}}{{{x^4} + {y^4}}} = 0$.
Therefore, $\mathop {\lim }\limits_{\left( {x,y} \right) \to \left( {0,0} \right)} f\left( {x,y} \right) = 0 = f\left( {0,0} \right)$.
By Definition of Continuity in Section 15.2 (page 768), $f\left( {x.y} \right)$ is continuous.
Case 2. $2p - 4 \le 0$
In this case $p \le 2$.
Consider $p=2$. The limit (1) becomes
$\mathop {\lim }\limits_{r \to 0} {r^{2p - 4}}\frac{{{{\cos }^P}\theta {{\sin }^P}\theta }}{{{{\cos }^4}\theta + {{\sin }^4}\theta }} = \mathop {\lim }\limits_{r \to 0} \frac{{{{\cos }^P}\theta {{\sin }^P}\theta }}{{{{\cos }^4}\theta + {{\sin }^4}\theta }}$
Fixing $\theta = 0$, that is, approaching $\left( {0,0} \right)$ along the positive $x$-axis gives
$\mathop {\lim }\limits_{r \to 0} \frac{{{{\cos }^P}0{{\sin }^P}0}}{{{{\cos }^4}0 + {{\sin }^4}0}} = 0$
Fixing $\theta = \frac{\pi }{3}$, that is, approaching $\left( {0,0} \right)$ along the positive $y$-axis gives
$\mathop {\lim }\limits_{r \to 0} \frac{{{{\cos }^P}\pi /3{{\sin }^P}\pi /3}}{{{{\cos }^4}\pi /3 + {{\sin }^4}\pi /3}} = \frac{{{2^{3 - 2P}}\cdot{3^{P/2}}}}{5}$
Since different values of $\theta$ give different results, the limit (1) does not exist.
Consider $p<2$.
As $r \to 0$, ${r^{2p - 4}} \to \infty $. The limit (1) becomes
$\mathop {\lim }\limits_{r \to 0} {r^{2p - 4}}\frac{{{{\cos }^P}\theta {{\sin }^P}\theta }}{{{{\cos }^4}\theta + {{\sin }^4}\theta }} = \infty $
So, the limit (1) does not exist.
Thus, if $p \le 2$, the limit (1) does not exist. Hence, $f\left( {x,y} \right)$ is discontinuous at $\left( {0,0} \right)$.
From the results in Case 1 and Case 2, we conclude that $f\left( {x,y} \right)$ is continuous at all $\left( {x,y} \right)$ if $p>2$ but is discontinuous at $\left( {0,0} \right)$ if $p \le 2$.
