Answer
.167L
Work Step by Step
We know that the center of mass of the overall system must be, at most, right over the edge of the table, or the entire set of books will fall. Thus, we call x the distance the last book hangs over the table to find:
$3x = .25L+x+.5L-.25L-x\\ 3x = .5 \\ x=.167L$