Answer
Take the shell's motion to be the positive side; the shell's velocity is $+547m/s$
Work Step by Step
In the 1st case, only the shell moves, so the work done by gunpowder on the shell is $$W=\frac{1}{2}m(v_f^2-v_0^2)=\frac{1}{2}\times85(551^2-0)=1.29\times10^7J$$
As stated in the exercise, the gunpowder imparts the same KE to the system, meaning in the 2nd case, the amount of work $W$ done on the system is the same, equaling $1.29\times10^7J$
In the 2nd case, both the shell and the cannon move, so taking the cannon with mass $M=5.8\times10^3kg$ and velocity $V$ and the shell with mass $m=85kg$ and velocity $v$, we have $$W=\frac{1}{2}M(V_f^2-V_0^2)+\frac{1}{2}m(v_f^2-v_0^2)$$
Both the cannon and the shell start from rest, so $v_0=V_0=0$
$$W=\frac{1}{2}M(V_f^2)+\frac{1}{2}m(v_f^2)$$ $$(2.9\times10^3)V_f^2+42.5v_f^2=1.29\times10^7 (1)$$
Also, according to the conservation of linear momentum, $$MV_f+mv_f=MV_0+mv_0=0$$ $$(5.8\times10^3)V_f+85v_f=0 (2)$$
Solving (1) and (2), we get $\vec{v}_f=+547m/s$ (take the shell's motion to be the positive side).
