I have never compared both techniques. This book may answer your question.

https://books.google.co.uk/books?hl=en&lr=&id=FlndBwAAQBAJ&oi=fnd&pg=PA1&ots=KUFKErK_Qa&sig=_pQGhlNVHdcBq0a9XNwytgPL9o4#v=onepage&q&f=false

In the last few years, hybrid metaheuristic is discipline that has been rising. It uses different concept and philosophy used in Intelligent system to create new non-deterministic algorithms. Iterated local search, memetic algorithms are example of them. 

Neural network have been evolved by CGP successfully. The results were positive with robotic related problems so far. Deep learning has been used for a while, but I would need to stand corrected that they have been optimised with a form of GA or GP.

It would be interested to have an answer to this question in many domain. I wonder if it would be even more interesting to find out how both techniques could outperform both GA (or GP) as well as ANNs.

I have never compared both techniques. This book may answer your question.

https://books.google.co.uk/books?hl=en&lr=&id=FlndBwAAQBAJ&oi=fnd&pg=PA1&ots=KUFKErK_Qa&sig=_pQGhlNVHdcBq0a9XNwytgPL9o4#v=onepage&q&f=false

In the last few years, hybrid metaheuristic is discipline that has been rising. It uses different concept and philosophy used in Intelligent system to create new non-deterministic algorithms. Iterated local search, memetic algorithms are example of them. 

Neural network have been evolved by CGP successfully. The results were positive with robotic related problems so far. Deep learning has been used for a while, but I would need to stand corrected that they have been optimised with a form of GA or GP.

It would be interested to have an answer to this question in many domain. I wonder if it would be even more interesting to find out how both techniques could outperform both GA (or GP) as well as ANNs.

Genetic Programming does not use only tree. Graph-based GP is only one part of this huge branch of artificial intelligence.  Example of Graph based GP is PDGP, CGP and GRAPE. These have built a graph using some genetic operators. 

From my research, it is clear that a tree bloat and algorithms can grow very large. in a genetic improvement context, it is very effective.  In hyper-heuristics and supervised machine learning, CGP can evolve mathematical expression as well as algorithms  in a much better expressive way. They tend to be shorter and easier to understand their logic. 

It would interesting if a Neural network would work too in this aspect. 

"ANN can be considered as "special instance" of GP algorithms"

Can we say that?

These are two different things:

A GA is an optimiser (and GP is an optimiser that can have operators as operands);

An ANN is a functional approximator, just like a nonlinear power series.

They both "learn", but an ANN needs an optimiser to do it.

See also http://userweb.eng.gla.ac.uk/yun.li/ga_demo/

Thank you Amir for your answer.

But i find a lot of "certainty" in what you say and this may end this discussion prematurely :)

As you said, ANN tries to approximate the unknown generator function of the mapping (input, output) with a linear/non-linear combination of predefined actions functions. 

GP is familiar with this type of problems. It is even more general than ANN. It can build complex programs (like trees, mathematical functions, if...then...else rules, etc...) that match any "input" to "any" output.

As you mentioned in your comment, Genetic algorithms are mainly used as optimizers. This should not discourage us to compare them to ANNs. If you take a parameterized function (i.e: f(X) = F_(a1, a2, ..., an)(X)), GAs can find the near optimal parameters (a1,...,an) that best fit X(input) to Y(output).

That's why i don't agree with what you said:

"As a result, we can't compare these two algorithms because they are totally different and they are supposed to carry out different aspects of machine learning problems."

I still think that GA and GP based algorithms can do more than what ANNs do. It strongly depends on the search space structure and the fitness function used.

Both GA and ANN-learning optimize. Moreover, there are multiple types of ANN-learning, including stochastic learning (which underlies GA). Isn't it curious that the most sophisticated "function" ever built from scratch (e.g., human adults) require both GA and ANN-like optimization? So, the evolution has came to the conclusion that GA alone is not sufficient alone, and has created ANN-like learning on top of it (at least as far the scope of the evolution on this Earth is concerned). Moreover, the ANN-alone seems implausible; they need hardware support. So, it's not about out-performance in general, but in specific "tasks", limits of the underlying hardware, time- and energy-constraints, etc.

At least, my friend.

You could mix then and take profit of both.

For instance, you could easily apply metaheuristics based algorithms to adapt and select the "best" set of inputs for your ANN.

"Deep networks" that learn to represent distributions of data make feature extraction very easy. See for example Hinton et al, Science 2006; I used this method for efficient encoding of a large amount of visual sets - the trained network learned that one efficient way to encode this 900-dimensional data is to extract the numerosity of the sets.

I would think that stochastic GA optimization could be useful in situations in which the complexity of the computational problem is too high; highly non-linear, etc., so ANNs can not resolve it in gradual multi-step (layer-wise) fashion and would need to resort to single-neuron-per-single-case (the universal learnability of the back-prop learning algorithm).

The answer to the question depends on which NN you are referring to and what task you are referring to. Traditional BP NN can get stuck on local minima without optimization. One such optimization  is to use a optimization technique such as annealing or GA at the initial stage to jump out of possible local minima. This has been demonstrated to be very useful ways on improving BP NN performance. In terms of comparison between BP and GA it was shown early on(90's) that there was an advantage to using GA over BP.  In terms  of comparison with other NN such as restricted Boltzmann machines, I personally have not seen any papers(if someone has, please post reference). What I would find challenging is to achieve higher generalization and time efficiency with GA as opposed to CD.

Alternatively, GP can be used for classification and approximation as does the NN and these can be compared for performance side by side. Again the results were done with  RPROP and not with restricted Boltzmann machines in a deep architecture.

Does Indian cuisine outperform Chinese cuisine? Or Italian?

It all depends on respective cooks, specific dish, and individual tastes of consumers, right?

Such wide sweeping generalizations of "is X better than Y" don't make sense, because the only honest answer you can get is "it depends". Same applies to modeling methods. I remember publications boasting "SVM outperformed ANN!". Well, we took data sets from this paper and ran through our implementation of SVM and ANN algorithms. And no, the results showed something opposite - ANN models were better than SVM, albeit by not much. But we don't generalize on the basis of this example, because the net outcome is not just the matter of algorithm ("cooking recipe"), but much more strongly depends on software implementation ("the cook"), particular data sets ("dishes"), and test cases ("consumers").

Now, to see how implementation influences modeling results you may want to check this paper:

http://onlinelibrary.wiley.com/doi/10.1002/jps.21494/abstract

Forget methods as complex as machine learning algorithms, in there you will find a very simple, deterministic equation called "MlogP", which stands for Moriguchi method of computing logP (the octanol/water partition coefficient - https://www.jstage.jst.go.jp/article/cpb1958/40/1/40_1_127/_pdf). You might think that processing such a simple equation should not depend on implementation and yet the results on Mannhold et al show dramatic differences between our version MlogP and the MlogP implemented by another software vendor. A simple equation! Authors notice and discuss this case.

I tried from relatively long time to do complex operations like symbolic regression using ANN only, I couldn't do that at this time without the use of GA (I have attached it). However; recently I am working on that again and found an algorithm that makes symbolic regression using ANN only and without the need for GA.

I will send you a copy of the paper once I publish it.

Article Evolving comprehensible neural network trees using genetic algorithms

You can even combine both techniques evolving the neural network. I think difficulty depends on the constraints each model have: in neural networks to determine the number of layers and networks and in genetic algorithm selection of population size, genetic operators and fitness function.

depends on area of application, as for time series analysis definitely ANN outperforms GA.

Your question about GA (or GP) outperforming NN deeply interested me. I had the same interrogation. Especially about GP, because of its graph representation and the fact it recombines true mathematical elements.

My "feeling" (I'm not expert) is that GP have the advantage of a better "expressiveness" while ANNs are more efficient. That's why I wonder what could be the efficiency and expressiveness of an algorithm similar to GP, but reduced of its "population/individuals" paradigm while extended with a “inverse mutation” operator. I've made some "toy" experiments and I found it (empirically) efficient so far. (btw, any advice on such attempt or link to some similar work would be greatly appreciated)

About the NFL theorems, they talk about the set made by all problems (in short). Therefore, could we state that all AI problems form a small subset w.r.t. the latter, subset in which there is a lot of correlations (and not uniformly distributed), so, that, in this context (practice) NFL do not hold ?

For me, GP should be more efficient than ANN whatever its architecture.

The problem is essentially in the Learning Strategy used in GP !

That's why they are actually a little bit "wilder" than ANN.

This work totally relies on ANN.

The algorithm performance is compared with GP.

Article Multi-objective symbolic regression using long-term artifici...