I also do not uderstand the question It is not problem to write Hamiltonian: simply add two operators of kinetic energy The problem is how to solve the system of coupled nonlinear equations (in classical theory) or the differential equation in partial derivatives in quantum case
In classical physics the ground state energy would be given by the value of the potential energy for the values of x and y that correspond to its minimum value, i.e. p(x)=0, p(y)=0, 2x + 2x y2=0 and 2y + 2y x2=0, giving x=0 and y=0 and energy equal to 0, since y2+2, respectively x2+2, never vanishes.
To find the ground state energy, when quantum effects become relevant, one typically solves the Schrödinger equation. One can estimate the ground state energy by computing / = , where |ψ> is some linear combination of trial states (which comes down to the same thing).
One can, also, attempt to compute the ground state energy using perturbation theory about the ground state of the harmonic oscillator, treating the x2y2 term as a perturbation. One way might be to express the position operators in terms of the creation and annihilation operators of the harmonic oscillators and compute the expectation value of x2y2 in that vacuum. It won't vanish.
The expression given is that of the Hamiltonian, however, so the question seems strange.
Usually I do not use Matlab However all necessary formulas for calculation of general case of coupled oscillators by means of operator method are written in my book "Nonperturbative description of quantum systems" Springer 2015