Answer
Maximum value: $f(1,2)=4$
Minimum value: $f(2,4)=-64$
Work Step by Step
Second derivative test: Some noteworthy points to calculate the local minimum, local maximum and saddle point of $f$.
1. If $D(p,q)=f_{xx}(p,q)f_{yy}(p,q)-[f_{xy}(p,q)]^2 \gt 0$ and $f_{xx}(p,q)\gt 0$ , then $f(p,q)$ is a local minimum.
2.If $D(p,q)=f_{xx}(p,q)f_{yy}(p,q)-[f_{xy}(p,q)]^2 \gt 0$ and $f_{xx}(p,q)\lt 0$ , then $f(p,q)$ is a local maximum.
3. If $D(p,q)=f_{xx}(p,q)f_{yy}(p,q)-[f_{xy}(p,q)]^2 \lt 0$ , then $f(p,q)$ is a not a local minimum and local maximum or, a saddle point.
Critical point are: $(0,-2)$
For $(x,y)=(1,2)$
$D=1 \gt 0$ ; and $f_{xx} =-4\lt 0$
Thus, when $D(p,q)=f_{xx}(p,q)f_{yy}(p,q)-[f_{xy}(p,q)]^2 \lt 0$ and $f_{xx}(p,q)\gt 0$ , then $f(p,q)$ is a local maximum.
For $(x,y)=(2,4)$
$D=8 \gt 0$ ; and $f_{xx} =4\gt 0$
Thus, when $D(p,q)=f_{xx}(p,q)f_{yy}(p,q)-[f_{xy}(p,q)]^2 \gt 0$ and $f_{xx}(p,q)\gt 0$ , then $f(p,q)$ is a local minimum.
Now, $f(1,2)=4xy^2-x^2y^2-xy^3=4(1)(2)^2-(1)^2(2)^2-(1)(2)^3=4$
and $f(2,4)=4xy^2-x^2y^2-xy^3=4(2)(4)^2-(2)^2(4)^2-(2)(4)^3=-64$
Therefore, we have Maximum value: $f(1,2)=4$
Minimum value: $f(2,4)=-64$
