See our method in: N.D. Lagaros, M. Papadrakakis, “Applied soft computing for optimum design of structures”, Structural and Multidisciplinary Optimization, Vol 45, pp. 787-799, 2012. Page: 797. In particular, the simple yet effective, multiple linear segment penalty function is used in this study for handling the constraints. According to this technique if no violation is detected, then no penalty is imposed on the objective function. If any of the constraints is violated, a penalty, relative to the maximum degree of constraints’ violation, is applied to the objective function. On the other hand, the optimization procedure is terminated when the best value of the objective function in the last 20 generations remains unchanged.

See our method in: N.D. Lagaros, M. Papadrakakis, “Applied soft computing for optimum design of structures”, Structural and Multidisciplinary Optimization, Vol 45, pp. 787-799, 2012. Page: 797. In particular, the simple yet effective, multiple linear segment penalty function is used in this study for handling the constraints. According to this technique if no violation is detected, then no penalty is imposed on the objective function. If any of the constraints is violated, a penalty, relative to the maximum degree of constraints’ violation, is applied to the objective function. On the other hand, the optimization procedure is terminated when the best value of the objective function in the last 20 generations remains unchanged.

Several approaches for the solution of constrained nonlinear problems are available. The most successful is sequential quadratic programming (SQP). For more details, you can see the following books.

• GILL P.E., MURRAY W., WRIGHT M.H.: ‘Practical optimization’ , Academic Press, London, 1981.
• BAZARAA M.,SHERALI H.,SHETTY C.M. : ‘Nonlinear programming: theory and algorithms’, Wiley, 2006.

Hi. NFL Theorem- No method is best for solving different problems and instances. Experiment with different methods.

witout doubt, each process obey its dynamic. In certain cases it is possible express the huge parameters by an orthogonal basis functions. La guerre or Chebyshev first kind.

To reduce, the effort an hyperbolic function is an efficient tool.

Hi, it might be best to adopt dynamic penalty formulae to converge to feasible region. You can check our paper in computers and OR co authored with chandra pedamallu. We did not make very good progress with our dual simulated annealing approach. In another study we tested SQP embedded in branch and bound algorithm with interval arithmetic. You can check our JOGO paper authored by pedamallu and ozdamar.

You should specify better the optimisation problem you need to solve:

-does it involve real variables only, discrete variables only, or a mix?

- is it a continuous and differentiable problem?

-are you looking for a global optimum or could a local optimum suffice?

- are the objective and constraints linear, quadratic or general nonlinear?

- how long does it take to evaluate objectives and constraints?

- can you provide a good initial guess for every instance of the problem?

- do you know any reasonable upper or lower bound of the objective function?

- do you have inequality constraints, or equality constraints? In case of equality constraints, what kind of violation of the constraints can you consider acceptable?

All these factors, plus several others, are required to appropriately choose an optimisation algorithm.

I guess the problem you're trying to solve falls squarely into the Nonlinear Programming problems domain, so you should try some Interior Point or Sequential Quadratic Programming algorithm: these algorithms can satisfy nonlinear constrained optimisation problems, handle nonlinear equality and inequality constraints very tightly, and give you a certificate of local optimality. However, they require you to provide a reasonably good initial guess, the objectives and constraints should not include any discontinuty and they only converge to local optima. 200 to 1000 variables are nothing for these algorithms (it's possible to solve some well structured problems with 1M variables in a couple of hours on a laptop)

Thank you Dr. Ricciardi and Dr. Abderrahim

I think the SQP is best for my problem.

what I needed to know is that for 200 to 1000 parameters is it fast enough or not, where Dr. Ricciardi mentioned that it is also possible to solve it with up to 1M parameters.

Thank you Dr. Lagaros and Dr. Ozdamar

I really appreciate your suggestion, if SQP was not suitable for my problem I will try your suggestions.

Dear Dr. Dupin and Dr. Adamuthe

I do not know about NFL, and I appreciate that you introduce me such a valuable definition.

but I have read wiki and it seems that one interpretation of NFL is:

"a general-purpose universal optimization strategy is theoretically impossible, and the only way one strategy can outperform another is if it is specialized to the specific problem under consideration"

It seems that for a specific problem we can choose an optimization method among different methods, and I somehow define my problem; it is training of multi layer perceptron neural network with some constraints, and I think as Dr. Ricciardi said it falls into Nonlinear Programming problems domain. So we may have the chance to choose the best algorithm? dont you think so?

Dear Ali Namadchian,

The theory behind NLP solvers is that if the objectives and constraints are regular enough and you're not too far away from the optimal solution, you can expect quadratic convergence to the local optimum (like going to 1e-1 to 1e-8 in 4 iterations). These methods obviously benefit from the knowledge of the derivatives, but if you cannot provide them the algorithms perform a finite difference approximation, which slows down things a bit but still works quite well.

Just to give you an idea, I routinely solve well structured and well scaled fully nonlinear NLP problems with 200 to 2000 variables in seconds to minutes through Matlab's fmincon NLP solver, which definitely isn't the fastest option available (but it's fast enough and interfaces well with the rest of my work). If I have a very good guess that just needs some refinement, it can take a fraction of a second or even less. I stress the fact however that the problem has to be well structured and especially well scaled.

Regarding the chance of choosing the "best" algorithm, it depends dramatically on the problem formulation and characteristics: if your functions contain severe disconitnuities you cannot expect a gradient based optimiser to work well, for example. On the flip side, if it is differentiable everywhere but you use a derivative free method, it will converge much more slowly. Also, the choice of "best" algorithm is itself a kind of meta-optimisation with different objectives and constraints: are you looking for best runtime, best accuracy, robustness, least memory, least required knowledge about the problem, easiest usage... ? There's very likely no definitive answer.

I used the external penalty function in my work to optimize the problem with the simultaneous occurrence of continuous and discrete decision variables.

https://www.researchgate.net/publication/257699580_Application_of_mixed_algorithms_of_optimization_in_designing_of_big_power_low_voltage_induction_motors?_sg=sAwstBzta6_BunsfAYyqheM39m5fCLWSnT5WB2QldK20i02ApkTDgdp24TWvcQMsY_J_WqdxRDXm7VUSgyeQKJwjkCFeyFCbp8lEwyRb.6aL5KkBA_F14AeDyGPMZwvSwlwLlFe4Ce7J02VFwHm2Ny17QYvnew2Zts8EmmHSQKhRyZtj7UiENM16cr0YXoA

https://www.researchgate.net/publication/224394012_Optimization_of_big_power_low_voltage_induction_motor_using_Hybrid_Optimization_Algorithms?_sg=kS_VvQChQtidIbHBcdZy-YFk85fZIZD9Yl7OIgNvbWHNLXB1LH7TdaQqafRQ2gbRyWlmziNZPbbF-Bacr2MsMRHg5ijwCtjRSgRAZbn_.8MRy7kFZPo8cVeLy6CxIpL_u7-0OC3drKtkfQvTRfSYL1WLMhDEqYGebAQMgaDgSss72tg4jbgyfNAN_yX-5Fg

https://www.researchgate.net/publication/236216958_Influence_of_the_constraints_on_the_optimal_problem_solution_using_particle_swarm_algorithm?_sg=JnazpwquqZOnL24l5KzyVtYk4AL7v2SuH74eoKHM9-SJWlhq_LffUvlnvbVDUooNjLJgqdQj0IR7IeRWBzDLoLlSc9dFaVn1mYR9zO4H.TzJMY8mlag3MKe7wp3EbgxhGSEZbRdRqkiV9uB-RvqS-ezknEmqq2_H7B8eogZzlStcbvZqeBLwWryLFWvFpHQ

Adjoint based optimization methods are good for problems with large number of parameters. Regarding method for constrained optimzation, you can look into Quasi-Newton methods like L-BFGS-B algorithm.

You can use Cross-Entropy Method. See, please, on my page on the RG the article "The Cross-Entropy Method for Continuous Multi-Extremal Optimization".

Regards, Sergey.

I can suggest the Modified Subgradient Algorithm (MSG) and MSG based on feasible values (F-MSG) by Refail Gasimov (or Kasimbeyli) to solve nondifferentiable constrained optimization problems.

Best regards, Ozden

may this papers helps you with the most new challenges belongs to your question

Article Handling Dropout Probability Estimation in Convolution Neura...

Chapter How Meta-heuristic Algorithms Contribute to Deep Learning in...

Lorenzo A. Ricciardi nailed it, it really depends on the type of problem youre trying to solve. if it isnt near quadratic, then you want to use metaheuristics or derivative free methods, otherwise interior point and SQP methods are good enough.