Warning: System is inconsistent

No, the system isn't inconsistent; just do the math: Integrating the first condition gives that f(x,y)=(a/2)x^2 + c(y). From this one obtains an, apparent,  contradiction, since the derivative of this expression wrt y is a function only of y and can't be equal to bx; unless b=0, which means that c(y)=c(0)=c, is a constant.

If one started out by integrating wrt y, one would've found f(x,y)=bxy + d(x); the derivative of this expression wrt x would then give by + d'(x)=ax=>d'(x)=ax-by, which, once more, is consistent only with b=0. 

So the only solution is f(x,y)=(a/2)x^2+c. Standard, elementary, exercise in undergraduate calculus. A mystery what it's doing on such a forum...

Try some function whose mixed partials are not continuous, cf Clairaut's Thm.

with b=0

if f(x,y)=(a/2)x^2+bxy+C then fx= ax +by and fy=bx