Answer
$$\frac{{\pi \sqrt 3 }}{3} - \ln \left( 2 \right) - \frac{{{\pi ^2}}}{{18}}$$
Work Step by Step
$$\eqalign{
& \int_0^{\pi /3} {x{{\tan }^2}x} dx \cr
& = \int_0^{\pi /3} {x\left( {{{\sec }^2}x - 1} \right)} dx \cr
& \cr
& {\text{Using integration by parts method }} \cr
& \,\,\,\,\,{\text{Let }}u = x,\,\,\,\,du = dx,\,\,\,\,\,dv = \left( {{{\sec }^2}x - 1} \right)dx,\,\,\,\,v = \tan x - x \cr
& \cr
& {\text{Integration by parts then gives}} \cr
& = x\left( {\tan x - x} \right) - \int {\left( {\tan x - x} \right)dx} \cr
& = x\left( {\tan x - x} \right) - \int {\tan xdx} + \int {xdx} \cr
& = x\tan x - {x^2} + \ln \left| {\cos x} \right| + \frac{{{x^2}}}{2} + C \cr
& = x\tan x + \ln \left| {\cos x} \right| - \frac{{{x^2}}}{2} + C \cr
& {\text{then}}{\text{,}} \cr
& \int_0^{\pi /3} {x{{\tan }^2}x} dx = \left( {x\tan x + \ln \left| {\cos x} \right| - \frac{{{x^2}}}{2}} \right)_0^{\pi /3} \cr
& = \left( {\frac{\pi }{3}\tan \frac{\pi }{3} + \ln \left| {\cos \frac{\pi }{3}} \right| - \frac{1}{2}{{\left( {\frac{\pi }{3}} \right)}^2}} \right) - \left( 0 \right) \cr
& = \frac{{\pi \sqrt 3 }}{3} + \ln \left( {\frac{1}{2}} \right) - \frac{{{\pi ^2}}}{{18}} \cr
& = \frac{{\pi \sqrt 3 }}{3} - \ln \left( 2 \right) - \frac{{{\pi ^2}}}{{18}} \cr} $$
