According to the article, CPLEX has a specific solver for mixed quadratic optimization problems. Maybe this answer my question. Thank you, Mrs Parsapoor.

Hi

I advise you to see the document " OPTIMIZATION PROBLEM TYPES - LINEAR AND QUADRATIC PROGRAMMING ". 

You will find what you need.

Best regards

http://www.solver.com/linear-quadratic-programming

Integer programming with quadratic constraints is "worse" than NP-hard.  In 1973, Robert G. Jeroslow proved that it is "undecidable", which means that there cannot exist any algorithm that is guaranteed to terminate in finite time.

Nevertheless, in practice, most problems have bounded variables.  In that case, the problem becomes "merely" NP-hard.

You might find this survey paper useful:

S. Burer & A.N. Letchford (2012) Non-convex mixed-integer nonlinear programming: a survey. Surveys in Oper. Res. and Mgmt. Sci., 17(2), 97-106.

The difficulty of the problem also depends on whether the problem is a convex one or not.

CPLEX MIQP solver can solve convex quadratic problems (to a global optimum). For nonconvex problems  things are (much) more complicated.

CPLEX V12.6.0 is capable of solving a quadratic program (that is, a problem with linear constraints and one or more quadratic terms in the objective function) for which the space of optimal solutions is not convex to global optimality. Such problems are known as nonconvex QPs. In addition to the linear constraints and quadratic objective terms, the problem can also include integer constraints, and CPLEX V12.6.0 can solve the MIQP to global optimality.

Source: https://www.ibm.com/support/knowledgecenter/SSSA5P_12.6.3/ilog.odms.studio.help/CPLEX/ReleaseNotes/topics/releasenotes126/newNonconvexMIQP.html

You could use Gurobi, too (look for MIQCP in the documentation)

www.gurobi.com