Dear Dr. Dudek,

I agree that tuning the EAs with many parameters is difficult,tedious and problem dependent. Among the algorithms that I have worked with (DE, PSO, GA, CS, Imperialist competitive algorithms and FA), I prefer DE and the good old PSO.

DE (mentioned also above) has only two parameters (F and CR) if we fix number of population to 10 times of number of variables) and has a good performance.

Hi,

my prefered solution to this problem is to use an algorithm, which offers (1) many parameters to tune so that it can be adjusted to the optimization problem and which offers (2) a mechanism to tune itself, if I do not have additional information. For example:

R. Storn and K. Price, “Differential Evolution: A simple and efficient heuristic for global optimization over continuous spaces,” Journal of Global Optimization, vol. 11, pp. 341–359, 1997

is a good optimization algorithm, which can be tuned in various ways. Additionally, there exists an automatic, self-tuning version:

J. Brest, S. Greiner, B. Boskovic, M. Mernik, and V . Zumer, “Self-Adapting Control Parameters in Differential Evolution: A Comparative Study on Numerical Benchmark Problems,” IEEE Transactions on Evolutionary Computation, vol. 10, pp. 646–657, 2006.

If you want the simplest optimization algorithm that does not make use of any gradient-based information, I think you should turn your attention to the so-called "single state" algorithms, i.e., that does not track multiple hypotheses at the same time. In this sense, I believe two classical algorithms that are always worth a try are the Simulated Annealing and the Tabu Search (plus many of their variants):

http://en.wikipedia.org/wiki/Simulated_annealing

http://en.wikipedia.org/wiki/Tabu_search

In its simplest form, Tabu Search has only ONE free parameter, i.e., the size of the tabu list. Similarly, Simulated Annealing with a classical exponentially decreasing cooling schedule has two parameters. Contrast this with the simplest genetic algorithm, where you need to choose a population size, a reproduction rate, a mutation rate, a tournament size (if you use tournaments), etc., and all of them influence the final solution.

I fully agree with you, that tuning is important and difficult part of the optimization procedure. For my point of view, for Cross-Entropy algorithm the tuning is more simple and "physicall". See, please, article: The Cross-Entropy Method for Continuous Multi-Extremal Optimization.Dirk P. Kroese, Sergey Porotsky, Reuven Y. Rubinstein

Methodology And Computing In Applied Probability 08/2006; 8(3):383-407. ·

https://www.researchgate.net/profile/Sergey_Porotsky/publications

It depends on the type of problem which you are encountering. Some problems GA will give better solutions. Some problems ACO will give better solutions and another problems some other heuristics.

Finally my questions is which problem are you looking.

One more thing now a days if you have have data machine learning techniques are preferable.

Dear Dr. Dudek,

I agree that tuning the EAs with many parameters is difficult,tedious and problem dependent. Among the algorithms that I have worked with (DE, PSO, GA, CS, Imperialist competitive algorithms and FA), I prefer DE and the good old PSO.

DE (mentioned also above) has only two parameters (F and CR) if we fix number of population to 10 times of number of variables) and has a good performance.

The term "Heuristic” is normally associated with “Empirical”, and in these cases random search and fine tuning can be essential, due to the weakness of the application of the optimization techniques. Normally empiricism is not a good adviser. I would prefer terms found in the literature as “artificial intelligence”, “parallel computing” or “stochastic optimization” related with the evolutionary algorithms (as GA, with flexible codes adapted to the problems).

There's no such thing as a free lunch.

According to [1]: "The No Free Lunch Theorem of Optimization (NFLT) is an impossibility theorem telling us that a general-purpose universal optimization strategy is impossible, and the only way one strategy can outperform another is if it is specialized to the structure of the specific problem under consideration."

[1] Y.-C. Ho, D. L. Pepyne (2002). Simple Explanation of the No Free Lunch Theorem of Optimization. Cybernetics and Systems Analysis 38(2): 292-298.

Nevertheless I found PSO conceptually easier than other heuristic optimization techniques such as GA. Compared to GA, PSO does not have genetic operators like crossover and mutation, it has memory which is important to the algorithm, it takes real numbers as particles (no need to change to binary encoding), and it is very easy to implement with few parameters (w, C1 & C2) to adjust.

Of course that everyone knows that only very (very, very) simple GAs has binary code. The chromosomes (characteristics of each individual) and the genetic operators can be modelled by multiple (multiple, multiple) ways.

Jacques de la Palice would not say better.

Hands down, if you can make a greedy algorithm, you're in a very good place! The only difficulty is proving the correctness of such an algorithm for the problem you are interested in. Sometimes it isn't very easy.

Dear Daniel Page

With respect to your intelligent answer:

Adjectives excluded, I fully agree with you. The skepticism can always come. But if you can find the (absolute) hill where it is the absolute optimal solution (for example in a large multimodal space), it looks very easy for any current optimization technique (even heuristic) to climb that hill. The simplest one is a simple classic "hill climbing” (an algorithm that generates a solution in the vicinity and accepts the solution if and only if the solution is better than the previous one – a kind of selective/empirical mutation in a GA or a search with a system frozen in a simulated annealing).

The difficulty is understand the logic of the nature.

Best regards

There are some compromises. For example, recent evolutionary algorithms such as the CMA-ES [1] have parameters that can be tuned, but offer some default values to the user, that are usually satisfying with regards to the problem at hand. You can (almost) always improve the results or the speed of convergence by fine-tuning the default parameters, but in practice the default values work well in most cases.

[1] http://en.wikipedia.org/wiki/CMA-ES

Depending on the dimension of your problem and the amount of function evaluations you are willing to spend you could also use some DFO (derivative free optimization) algorithm like MCS, SNOBFIT, or LGO. There, the number of parameters is very small, and good default values are provided.

Thank you very much for your opinions.

Each of us has a collection of his favorite algorithms and some experience in their application and selection of their parameters.

Hermann Schichl mentioned about derivative free optimization. I have found an interesting work comparing 22 DFO algorithms:

http://link.springer.com/article/10.1007%2Fs10898-012-9951-y

Dear Grzegorz. Thank you for providing the link to the paper "Derivative-free optimization: a review of algorithms and comparison of software implementations" by Luis Miguel Rios and Nikolaos V. Sahinidis. It was quite state of art on derivative free optimization algorithms.

The simplest and easiest to use heuristic usually is a simple greedy method. Given the problem description and the option to find a feasible solution with it you might like to try improving it with a local search approach. And then you might like to use reactive tabu search (the reason for doing that is that it does not have the difficult/tedious parameterization as other methods). If you can identify subproblems that may be solved to optimality you might like to combine any heuristic or metaheuristic with mathematical programming deriving a matheuristic (like POPMUSIC or the corridor method).

PSO

Simplest and easiest to use...

If you have any difficulties in tuning the algorithm parameters, why don't you try the parameter-free optimization methods. Recently, some new optimization algorithms do not require the users to tune algorithm-specific algorithm. You just need only to tune common control parameters such as population size and maximum iterations.

You might check: Symbiotic Organisms Search (SOS). It was first proposed to solve continuous optimization problems. SOS is inspired by the symbiotic relationship phenomena (mutualism, commensalism, parasitism) that used by many organisms to survived in the ecosystem.

Here are the initial work of SOS and the MATLAB code invented by the authors.

Hope it can give you more alternative solutions to consider.

http://www.mathworks.com/matlabcentral/fileexchange/47465-sos-m

Article Symbiotic Organisms Search: A new metaheuristic optimization algorithm

It depends on the application. For several applications, I wouldn't rely on heuristics. Reproducibility of the results is a great concern.

A simple Particle Swarm Optimization variant can be coded in less than an hour for a moderate programmer (e.g. in Python/Matlab). Generally, swarm intelligence seems to provide good results for many types of problems.  Ant Colony Optimization is also a popular choice with many successful applications. It is a bit harder to program and control and needs some more programming experience.

It is advisable to test first on a problem the solution of which is known through a deterministic technique. Convergence is not always guaranteed.

Hi Grzegorz

I think there is no ready-to-use fine-tuned heuristic. Fine-tuning can be done only by experimenting, the larger the size of the instances the best the resulting heuristic.

Regards

Salim

I believe I have a metaheuristic which uses no parameters, and needs no fine tuning; check it here:

http://web.unbc.ca/~bluskovi/pdo.pdf

Journal of Combinatorial Optimization, 2015, DOI 10.1007/s10878-014-9826-x.

The final publication is available at link.springer.com; more precisely, here:

http://link.springer.com/article/10.1007/s10878-014-9826-x

You may try the following algorithms:

https://www.researchgate.net/publication/260288538_Discrete_size_optimization_of_steel_trusses_using_a_refined_big_bangbig_Crunch_Algorithm

 https://www.researchgate.net/publication/256918712_An_exponential_big_bang-big_crunch_algorithm_for_discrete_design_optimization_of_steel_frames

These optimization algorithms are easy to implement and have few parameters to set.

Best Wishes

Article Discrete size optimization of steel trusses using a refined ...

Article An exponential big bang-big crunch algorithm for discrete de...

Hi

In the following paper, a new intelligent method called SQPSO is proposed.

This technique requires just one control parameter to set its operation and also it is easy to be implemented.

Species-based Quantum Particle Swarm Optimization for economic load

dispatch,International Journal of Electrical Power & Energy Systems

Volume 63, December 2014, Pages 311–322

I recommend DE.

In my point of view, differential evolution maintains two intersting caracteristics (A) easy to implement and (B) good performance.