Answer
$B^2-4AC\lt0$
Work Step by Step
The general equation of a conic in the form of $Ax^2+Bxy+Cy^2+Dx+Ey+F=0$
(i) defines a parabola if $B^2-4AC=0$
(ii) defines an ellipse (or a circle) if $B^2-4AC\lt0$
(iii) defines a hyperbola if $B^2-4AC\gt0$
Hence here the answer is $B^2-4AC\lt0$
