Answer
$$\cos(A+B)=-\frac{16}{65}$$
Work Step by Step
$$\cos A=\frac{3}{5}\hspace{1cm}\sin B=-\frac{5}{13}\hspace{1cm}0\lt A\lt \frac{\pi}{2}\hspace{1cm}\pi\lt B\lt\frac{3\pi}{2}$$
Since $0\lt A\lt \frac{\pi}{2}$, we get that $A$ is in quadrant I, meaning $\sin A$ and $\cos A$ are positive.
Also, $\pi\lt B\lt\frac{3\pi}{2}$ means that $B$ is in quadrant III, and both $\sin B$ and $\cos B$ are negative.
To do this exercise, we need to know the exact values of $\sin A$, $\cos A$, $\sin B$ and $\cos B$. And we can figure them using Pythagorean Identities.
$$\sin^2 A=1-\cos^2 A=1-\Big(\frac{3}{5}\Big)^2=1-\frac{9}{25}=\frac{16}{25}$$
$$\sin A=\frac{4}{5}\hspace{1cm}(\sin A\gt0)$$
$$\cos^2 B=1-\sin^2 B=1-\Big(-\frac{5}{13}\Big)^2=1-\Big(\frac{25}{169}\Big)=\frac{144}{169}$$
$$\cos B=-\frac{12}{13}\hspace{1cm}(\cos B\lt0)$$
Now we can calculate $\cos(A+B)$.
According to the identity of cosine sum, we have
$$\cos(A+B)=\cos A\cos B-\sin A\sin B$$
$$\cos(A+B)=\frac{3}{5}\Big(-\frac{12}{13}\Big)-\frac{4}{5}\Big(-\frac{5}{13}\Big)$$
$$\cos(A+B)=-\frac{36}{65}+\frac{20}{65}$$
$$\cos(A+B)=-\frac{16}{65}$$
