Answer
$cos(tan^{-1}\frac{5}{12}-tan^{-1}\frac{3}{4}) = \frac{63}{65}$
Work Step by Step
Let $A = tan^{-1}\frac{5}{12}$
Then:
$cos~A = \frac{12}{\sqrt{5^2+12^2}} = \frac{12}{13}$
$sin~A = \frac{5}{\sqrt{5^2+12^2}} = \frac{5}{13}$
Let $B = tan^{-1}\frac{3}{4}$
$cos~B = \frac{4}{\sqrt{3^2+4^2}} = \frac{4}{5}$
$sin~B = \frac{3}{\sqrt{3^2+4^2}} = \frac{3}{5}$
We can find $~~cos(tan^{-1}\frac{5}{12}-tan^{-1}\frac{3}{4})$:
$cos(A-B) = cos~A~cos~B+sin~A~sin~B$
$cos(A-B) = (\frac{12}{13})~(\frac{4}{5})+(\frac{5}{13})~(\frac{3}{5})$
$cos(A-B) = \frac{48}{65}+\frac{15}{65}$
$cos(A-B) = \frac{63}{65}$
Therefore, $~~cos(tan^{-1}\frac{5}{12}-tan^{-1}\frac{3}{4}) = \frac{63}{65}$
