Answer
(a) for $\left( {x,y} \right) \ne \left( {0,0} \right)$:
${f_x}\left( {x,y} \right) = \frac{{y\left( {{x^4} + 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
${f_y}\left( {x,y} \right) = \frac{{x\left( {{x^4} - 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
(b) Using the limit definition of the partial derivative we get
${f_x}\left( {0,0} \right) = {f_y}\left( {0,0} \right) = 0$.
${f_{yx}}\left( {0,0} \right) = 1$
${f_{xy}}\left( {0,0} \right) = - 1$
Thus, ${f_{xy}}\left( {0,0} \right)$ both exist but are not equal.
(c) We show that for $\left( {x,y} \right) \ne \left( {0,0} \right)$:
${f_{xy}}\left( {x,y} \right) = {f_{yx}}\left( {x,y} \right) = \frac{{{x^6} + 9{x^4}{y^2} - 9{x^2}{y^4} - {y^6}}}{{{{\left( {{x^2} + {y^2}} \right)}^3}}}$
We show that $\mathop {\lim }\limits_{h \to 0} {f_{xy}}\left( {h,0} \right) \ne \mathop {\lim }\limits_{h \to 0} {f_{xy}}\left( {0,h} \right)$, hence by definition ${f_{xy}}\left( {x,y} \right) = {f_{yx}}\left( {x,y} \right)$ is not continuous at $\left( {0,0} \right)$.
(d) The Clairaut's Theorem requires that both ${f_{yx}}$ and ${f_{xy}}$ be continuous functions for all points in its domain. But as it has been shown in part (c) they are not continuous functions at $\left( {0,0} \right)$. Thus, it does not contradict the theorem.
Work Step by Step
(a) We have $f\left( {x,y} \right) = xy\frac{{{x^2} - {y^2}}}{{{x^2} + {y^2}}}$ for $\left( {x,y} \right) \ne \left( {0,0} \right)$ and $f\left( {0,0} \right) = 0$.
Write $f\left( {x,y} \right) = \frac{{{x^3}y - x{y^3}}}{{{x^2} + {y^2}}}$.
1. Find the derivative with respect to $x$:
${f_x} = \frac{{\left( {{x^2} + {y^2}} \right)\left( {3{x^2}y - {y^3}} \right) - \left( {{x^3}y - x{y^3}} \right)2x}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
$ = \frac{{3{x^4}y + 3{x^2}{y^3} - {x^2}{y^3} - {y^5} - 2{x^4}y + 2{x^2}{y^3}}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
$ = \frac{{{x^4}y + 4{x^2}{y^3} - {y^5}}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
Hence, ${f_x}\left( {x,y} \right) = \frac{{y\left( {{x^4} + 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
2. Find the derivative with respect to $y$:
${f_y} = \frac{{\left( {{x^2} + {y^2}} \right)\left( {{x^3} - 3x{y^2}} \right) - \left( {{x^3}y - x{y^3}} \right)2y}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
$ = \frac{{{x^5} + {x^3}{y^2} - 3{x^3}{y^2} - 3x{y^4} - 2{x^3}{y^2} + 2x{y^4}}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
$ = \frac{{{x^5} - 4{x^3}{y^2} - x{y^4}}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
Hence, ${f_y}\left( {x,y} \right) = \frac{{x\left( {{x^4} - 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
In summary, for $\left( {x,y} \right) \ne \left( {0,0} \right)$:
${f_x}\left( {x,y} \right) = \frac{{y\left( {{x^4} + 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
${f_y}\left( {x,y} \right) = \frac{{x\left( {{x^4} - 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
(b) We have $f\left( {0,0} \right) = 0$.
Using the limit definition of the partial derivative we get
${f_x}\left( {0,0} \right) = \mathop {\lim }\limits_{h \to 0} \frac{{f\left( {h,0} \right) - f\left( {0,0} \right)}}{h} = \mathop {\lim }\limits_{h \to 0} \frac{{f\left( {h,0} \right)}}{h} = 0$
Similarly,
${f_y}\left( {0,0} \right) = \mathop {\lim }\limits_{k \to 0} \frac{{f\left( {0,k} \right) - f\left( {0,0} \right)}}{k} = \mathop {\lim }\limits_{k \to 0} \frac{{f\left( {0,k} \right)}}{k} = 0$
Thus, ${f_x}\left( {0,0} \right) = {f_y}\left( {0,0} \right) = 0$.
From part (a) we obtain for $\left( {x,y} \right) \ne \left( {0,0} \right)$:
${f_x}\left( {x,y} \right) = \frac{{y\left( {{x^4} + 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
${f_y}\left( {x,y} \right) = \frac{{x\left( {{x^4} - 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
Using the limit definition of the partial derivative we get the second derivatives of $f$:
${f_{yx}}\left( {0,0} \right) = \mathop {\lim }\limits_{h \to 0} \frac{{{f_y}\left( {h,0} \right) - {f_y}\left( {0,0} \right)}}{h}$
$ = \mathop {\lim }\limits_{h \to 0} \frac{{{f_y}\left( {h,0} \right)}}{h} = \mathop {\lim }\limits_{h \to 0} \frac{{{h^5}/{h^4}}}{h}$
$ = 1$
${f_{xy}}\left( {0,0} \right) = \mathop {\lim }\limits_{k \to 0} \frac{{{f_x}\left( {0,k} \right) - {f_x}\left( {0,0} \right)}}{k}$
$ = \mathop {\lim }\limits_{k \to 0} \frac{{{f_x}\left( {0,k} \right)}}{k} = \mathop {\lim }\limits_{k \to 0} \frac{{ - {k^5}/{k^4}}}{k}$
$ = - 1$
Hence, we conclude that ${f_{yx}}\left( {0,0} \right)$ and ${f_{xy}}\left( {0,0} \right)$ both exist but are not equal.
(c) Recall from part (a), we have for $\left( {x,y} \right) \ne \left( {0,0} \right)$:
${f_x}\left( {x,y} \right) = \frac{{y\left( {{x^4} + 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}} = \frac{{{x^4}y + 4{x^2}{y^3} - {y^5}}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
${f_y}\left( {x,y} \right) = \frac{{x\left( {{x^4} - 4{x^2}{y^2} - {y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}} = \frac{{{x^5} - 4{x^3}{y^2} - x{y^4}}}{{{{\left( {{x^2} + {y^2}} \right)}^2}}}$
1. Take the derivative of ${f_x}$ with respect to $y$ gives
${f_{xy}} = \frac{{{{\left( {{x^2} + {y^2}} \right)}^2}\left( {{x^4} + 12{x^2}{y^2} - 5{y^4}} \right) - \left( {{x^4}y + 4{x^2}{y^3} - {y^5}} \right)\left( {4y} \right)\left( {{x^2} + {y^2}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^4}}}$
$ = \frac{{\left( {{x^2} + {y^2}} \right)\left( {{x^4} + 12{x^2}{y^2} - 5{y^4}} \right) - 4y\left( {{x^4}y + 4{x^2}{y^3} - {y^5}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^3}}}$
${f_{xy}} = \frac{{{x^6} + 9{x^4}{y^2} - 9{x^2}{y^4} - {y^6}}}{{{{\left( {{x^2} + {y^2}} \right)}^3}}}$
2. Take the derivative of Subscript[f, y] with respect to $x$ gives
${f_{yx}} = \frac{{{{\left( {{x^2} + {y^2}} \right)}^2}\left( {5{x^4} - 12{x^2}{y^2} - {y^4}} \right) - \left( {{x^5} - 4{x^3}{y^2} - x{y^4}} \right)\left( {4x} \right)\left( {{x^2} + {y^2}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^4}}}$
$ = \frac{{\left( {{x^2} + {y^2}} \right)\left( {5{x^4} - 12{x^2}{y^2} - {y^4}} \right) - 4x\left( {{x^5} - 4{x^3}{y^2} - x{y^4}} \right)}}{{{{\left( {{x^2} + {y^2}} \right)}^3}}}$
${f_{yx}} = \frac{{{x^6} + 9{x^4}{y^2} - 9{x^2}{y^4} - {y^6}}}{{{{\left( {{x^2} + {y^2}} \right)}^3}}}$
Hence,
${f_{xy}}\left( {x,y} \right) = {f_{yx}}\left( {x,y} \right) = \frac{{{x^6} + 9{x^4}{y^2} - 9{x^2}{y^4} - {y^6}}}{{{{\left( {{x^2} + {y^2}} \right)}^3}}}$
Next, we evaluate $\mathop {\lim }\limits_{h \to 0} {f_{xy}}\left( {h,0} \right)$ along the line $y=0$, and $\mathop {\lim }\limits_{h \to 0} {f_{xy}}\left( {0,h} \right)$ along the line $x=0$:
$\mathop {\lim }\limits_{h \to 0} {f_{xy}}\left( {h,0} \right) = \mathop {\lim }\limits_{h \to 0} \frac{{{h^6}}}{{{h^6}}} = \mathop {\lim }\limits_{h \to 0} 1 = 1$
$\mathop {\lim }\limits_{h \to 0} {f_{xy}}\left( {0,h} \right) = \mathop {\lim }\limits_{h \to 0} \frac{{ - {h^6}}}{{{h^6}}} = \mathop {\lim }\limits_{h \to 0} \left( { - 1} \right) = - 1$
Since $\mathop {\lim }\limits_{h \to 0} {f_{xy}}\left( {h,0} \right) \ne \mathop {\lim }\limits_{h \to 0} {f_{xy}}\left( {0,h} \right)$, by definition ${f_{xy}}$ is not continuous at $\left( {0,0} \right)$.
Since ${f_{xy}}\left( {x,y} \right) = {f_{yx}}\left( {x,y} \right)$, so we conclude that ${f_{yx}}$ is also not continuous at $\left( {0,0} \right)$.
(d) In part (b) we show that the first derivatives of $f$ are equal, that is, ${f_x}\left( {0,0} \right) = {f_y}\left( {0,0} \right) = 0$. However, the second mixed derivatives ${f_{yx}}\left( {0,0} \right)$ and ${f_{xy}}\left( {0,0} \right)$ both exist but are not equal.
This does not contradict Clairaut's Theorem because the theorem requires that both ${f_{yx}}$ and ${f_{xy}}$ be continuous functions for all points in its domain. But as it has been shown in part (c) they are not continuous functions at $\left( {0,0} \right)$.
