Answer
$(b-a)\pi R^2$
Work Step by Step
Given function is $\vec{v} = ay\hat{x} + bx\hat{y} $ where $a$ and $b$ are constants.
The path given is a circular path of radius $R$ in the $xy$ plane
Stokes' Theorem states that
$\int_S (\vec{\nabla} \times \vec{v})\cdot \vec{da}= \oint_C \vec{v}\cdot \vec{dl} $
In Cartesian coordinate System $\vec{da} = dxdy\hat{z}$ and $\vec{dl} = dx\hat{x} + dy\hat{y}$
$ (\vec{\nabla} \times \vec{v})=
\begin{vmatrix}
\hat{x} & \hat{y} & \hat{z}\\
\frac{\partial}{\partial x} & \frac{\partial}{\partial y} & \frac{\partial}{\partial z}\\
ay & bx & 0
\end{vmatrix}$
$\qquad \quad \space =
\hat{x}(\frac{\partial}{\partial{y}} (0)- \frac{\partial}{\partial{z}} (x)) - \hat{y}(\frac{\partial}{\partial{x}} (0) - \frac{\partial}{\partial{z}} (ay))+\hat{z}(\frac{\partial}{\partial{x}} (bx) - \frac{\partial}{\partial{y}} (ay))$
$\qquad \quad \space = \hat{z}(b-a)$
And hence
$\int_S (\vec{\nabla} \times \vec{v})\cdot \vec{da}=\int_S (b-a)\hat{z}\cdot dxdy\hat{z}$
Since $(b-a)$ is constant
$\int_S (\vec{\nabla} \times \vec{v})\cdot \vec{da}=(b-a)\int_S dxdy$
For the given surface $\int_S dxdy = \pi R^2$
Hence,
$\int_S (\vec{\nabla} \times \vec{v})\cdot \vec{da}=\pi R^2 \qquad ...(i)$
$\oint_C \vec{v}.\vec{dl} = \oint_C (ay\hat{x} + bx\hat{y})\cdot (dx\hat{x}+dy\hat{y})$
$\oint_C \vec{v}.\vec{dl} = \oint_C (aydx + bxdy)$
For computing the line integral, Let's divide the circle in upper half $C_1$ and lower half $C_2$.
For upper half, $y = \sqrt{R^2-x^2}$ and for lower half $y = -\sqrt{R^2-x^2}$
Let's compute the line integral for upper half first,
$dy = \frac{1}{2\sqrt{R^2-x^2}}(-2x)dx = \frac{-x}{\sqrt{R^2-x^2}}dx$
So,
$\int_{C_1}\vec{v}.\vec{dl}=\int_{C_1} (a\sqrt{R^2-x^2}dx+bx(\frac{-x}{\sqrt{R^2-x^2}})dx)$
$\int_{C_1}\vec{v}.\vec{dl}=\int_{C_1} \frac{(a(R^2-x^2)-{bx^2})}{\sqrt{R^2-x^2}}dx$
$\int_{C_1}\vec{v}.\vec{dl}=\int_{C_1} \frac{aR^2-(a+b)x^2}{\sqrt{R^2-x^2}}dx$
Since the Counterclockwise direction is considered to be positive,
$\therefore x $ varies from $R$ to $-R$.
$\int_{C_1}\vec{v}.\vec{dl}=\int_{R}^{-R} (\frac{aR^2}{\sqrt{R^2-x^2}} - \frac{(a+b)x^2}{\sqrt{R^2-x^2}} )dx$
$\int_{C_1}\vec{v}.\vec{dl}=\big\{ {aR^2}.sin^{-1}({x\over R}) - (a+b)\big [{-x\over 2} \sqrt{R^2-x^2}+{R^2\over 2} sin^{-1}({x\over R})\big ]\big \}_R^{-R}$
$\int_{C_1}\vec{v}.\vec{dl}=\big\{ \frac{1}{2}(a-b)R^2sin^{-1}({x\over R})\big \}_R^{-R} \qquad \because \int_R^{-R} \frac{-x}{2}\sqrt{R^2-x^2}=0$
$\int_{C_1}\vec{v}.\vec{dl}=\frac{1}{2}R^2(a-b)sin^{-1}({-R\over R}) - \frac{1}{2}R^2(a-b)sin^{-1}({R\over R})$
$\int_{C_1}\vec{v}.\vec{dl}=\frac{1}{2}R^2(a-b)({-\pi \over 2}-{\pi \over 2})$
$\int_{C_1}\vec{v}.\vec{dl}=\frac{1}{2}R^2(b-a)\pi$
For the second integral,
$dy = -\frac{1}{2\sqrt{R^2-x^2}}(-2x)dx = \frac{x}{\sqrt{R^2-x^2}}dx$
$\int_{C_2}\vec{v}.\vec{dl}=\int_{C_2} (a(-\sqrt{R^2-x^2})dx+bx(\frac{x}{\sqrt{R^2-x^2}})dx)$
$\int_{C_2}\vec{v}.\vec{dl}=\int_{C_2} \frac{-aR^2+(a+b)x^2}{\sqrt{R^2-x^2}}dx$
Here $x$ varies from $-R$ to $R$
and hence
$\int_{C_2}\vec{v}.\vec{dl}=\int_{-R}^{R} \frac{-aR^2+(a+b)x^2}{\sqrt{R^2-x^2}}dx$
Reversing the limits we get,
$\int_{C_2}\vec{v}.\vec{dl}=\int_{R}^{-R} \frac{aR^2-(a+b)x^2}{\sqrt{R^2-x^2}}dx = \int_{C_1}\vec{v}.\vec{dl}$
and hence
$\oint_C \vec{v}\cdot \vec{dl} =\frac{1}{2}R^2(b-a)\pi + \frac{1}{2}R^2(b-a)\pi $
$\oint_C \vec{v}\cdot \vec{dl} =\pi R^2(b-a)$
Hence stokes theorem is proved.