Answer
$\mathrm{S}\mathrm{S}\mathrm{E}= 86.56$
(a) provides a better fit
Work Step by Step
Residuals and Sum-of-Squares Error (SSE)
If we model a collection of data $(x_{1}, y_{1})$ , . , $(x_{n}, y_{n})$ with a linear equation $\hat{y}=mx+b$,
then the residuals are the $n$ quantities (Observed Value-Predicted Value):
$(y_{1}-\hat{y}_{1}), (y_{2}-\hat{y}_{2}), \ldots, (y_{n}-\hat{y}_{n})$ .
The sum-of-squares error (SSE) is the sum of the squares of the residuals:
SSE $=(y_{1}-\hat{y}_{1})^{2}+(y_{2}-\hat{y}_{2})^{2}+\cdots+(y_{n}-\hat{y}_{n})^{2}.$
The model with smaller SSE gives the better fit.
----
(b)
Build a table,(the table below was generated in Excel)
\begin{array}{|cc|c|c|c|cc|}
\hline & x & y & y'=-0.2x+6 & (y-y') & (y-y') ^2 \\
\hline & 2 & 4 & 5.6 & -1.6 & 2.56 \\
& 6 & 8 & 4.8 & 3.2 & 10.24 \\
& 8 & 12 & 4.4 & 7.6 & 57.76 \\
& 10 & 0 & 4 & -4 & 16 \\
\hline & & & & {\bf SSE}= & {\bf 86.56} \\\hline
\end{array}
Case (a), where SSE was 80.04, provides a better fit .