As you mentioned you have done phasor estimation of voltage and current. I hope next step should be of finding impedance measured by relay by using phase and ground distance elements. If the measured impedance (R and X cordinates on impedance plane) lies inside your quadrilateral zone, then develop a logic that it should generate 1 or 0 as output (as per required to open breaker).