Answer
$h=\sqrt{a^2+ab}$.
Work Step by Step
Step 1. Using the figure given in the exercise, we have
$tan\theta=\frac{a}{h}$ and $tan(\theta+\beta)=\frac{a+b}{h}$.
Step 2. Since
$tan(\theta+\beta)=\frac{tan\theta+tan\beta}{1-tan(\theta) tan(\beta)}=\frac{a/h+tan\beta}{1-a\ tan(\beta)/h}=\frac{a+b}{h}$.
We have $tan\beta=\frac{bh}{h^2+a^2+ab}$.
Step 3. Assuming $0\lt \beta \lt \pi/2$, to maximize $\beta$ would be the same as to maximize $tan\beta$.
Step 4. Letting $f(h)=\frac{bh}{h^2+a^2+ab}$, we can set $f’=0$ and get $b(h^2+a^2+ab)-2bh^2=0$, which gives $h=\sqrt{a^2+ab}$.
Step 5. We can confirm that the $h$ value gives a maximum of $f$ by testing sign changes of $f’$ as $..(+)..(f’(h))..(-)..$
