If it was a differential equaiton, we should have a solution of the form y=f(x) or x = g(y)

If y = x, the dy/dx = 1 and dx/dy = 1. This seems to be a trival solution for the equation. It may be the only solution?

Your question boils down to, "Why does the AI give the wrong answer?". My understanding of the large language models is that they are very complicated neural networks, and that even their teams of inventors may disagree about their characteristics and capabilities. Some experts even have attributed sentience to AI. Neural networks assign weights to parameters, which are used in complicated algorithms, as you note. The weights are not visible to users. The compiled algorithms that make up the neural networks are deeply removed from human eyes by many levels of machine code. In the old days we used to say GIGO, or "garbage in, garbage out" if a program provided an incorrect result. Today, this does not hold true with A.I. Bad inputs can produce good outputs and good inputs can produce incorrect outputs. Correct information has become a matter of statistics.

So, you have dy/dx = dx/dy. ​This is a very simple ordinary differential equation.  Multiplying both sides of this equation by dy/dx, we get ​the ordinary differential equation: ​(dy/dx)^2 = 1. This is basically an elementary NON-LINEAR ODE that can be separated out to yield two LINEAR parts, as follows. Taking the square root of both sides of this second ODE gives dy/dx = +1 and dy/dx = -1. Integrating each of these new LINEAR ODEs with respect to x gives y = x + C and y = - x + C, respectively. One can see from these two resulting solutions that they are equations of a family of straight, diagonal, lines, some with a positive slope represented by y = x + C and some with a negative slope represented by y = -x + C. Hope that this helps.