Answer
$$\frac{{18x - 5}}{{36}}\sqrt {5x - 9{x^2}} + \frac{{25}}{{216}}{\sin ^{ - 1}}\left( {\frac{{18x - 5}}{5}} \right) + C$$
Work Step by Step
$$\eqalign{
& \int {\sqrt {5x - 9{x^2}} dx} \cr
& {\text{write the integrand as}} \cr
& = \int {\sqrt {5x - {{\left( {3x} \right)}^2}} dx} \cr
& {\text{Make an appropiate }}u{\text{ - substitution }} \cr
& u = 3x,\,\,\,\,\,\,\,du = 3dx,\,\,\,\,\,\,dx = \frac{{du}}{3} \cr
& {\text{write in terms of }}u \cr
& \int {\sqrt {5x - {{\left( {3x} \right)}^2}} dx} = \int {\sqrt {5\left( {\frac{u}{3}} \right) - {u^2}} \left( {\frac{{du}}{3}} \right)} \cr
& = \frac{1}{3}\int {\sqrt {\frac{5}{3}u - {u^2}} du} \cr
& = \frac{1}{3}\int {\sqrt {2\left( {\frac{5}{6}u} \right) - {u^2}} du} \cr
& {\text{Use the Endpaper Integral Table to evaluate the integral}} \cr
& {\text{By formula 112}} \cr
& \left( {112} \right):\,\,\,\,\,\,\int {\sqrt {2au - {u^2}} = } \frac{{u - a}}{2}\sqrt {2au - {u^2}} + \frac{{{a^2}}}{2}{\sin ^{ - 1}}\left( {\frac{{u - a}}{a}} \right) + C \cr
& {\text{take }}a = \frac{5}{6},\,\, \cr
& \frac{1}{3}\int {\sqrt {2\left( {\frac{5}{6}u} \right) - {u^2}} du} = \frac{{u - 5/6}}{6}\sqrt {2\left( {\frac{5}{6}u} \right) - {u^2}} + \frac{{{{\left( {5/6} \right)}^2}}}{6}{\sin ^{ - 1}}\left( {\frac{{u - 5/6}}{{5/6}}} \right) + C \cr
& {\text{simplifying}} \cr
& = \frac{{6u - 5}}{{36}}\sqrt {\frac{5}{3}u - {u^2}} + \frac{{25}}{{216}}{\sin ^{ - 1}}\left( {\frac{{6u - 5}}{5}} \right) + C \cr
& {\text{write in terms of }}x{\text{; replace }}3x{\text{ for }}u \cr
& = \frac{{6\left( {3x} \right) - 5}}{{36}}\sqrt {\frac{5}{3}\left( {3x} \right) - {{\left( {3x} \right)}^2}} + \frac{{25}}{{216}}{\sin ^{ - 1}}\left( {\frac{{6\left( {3x} \right) - 5}}{5}} \right) + C \cr
& = \frac{{18x - 5}}{{36}}\sqrt {5x - 9{x^2}} + \frac{{25}}{{216}}{\sin ^{ - 1}}\left( {\frac{{18x - 5}}{5}} \right) + C \cr} $$
