I would think that there are two reasons why they perform well:

1)The use of multiple layers have been known to perform better abstractions for a long time ( see for example An Introduction to Computing with Neural Nets by Lippmann where on figure 14 explains the types of decision regions that can be obtained from adding layers).

2) It allows the creation choke points that forces the neural network to generalize( see Reducing the Dimensionality of Data with Neural Networks by Hinton and Salakhutdinov)

These two reasons may allow you abstract to higher level representations. The problem has been that techniques used on shallow networks such as backpropagation perform badly on such networks, therefore historically limiting research on multiple hidden layers.

There is still much research to be done on deep learning to explain their performance.

I would think that there are two reasons why they perform well:

1)The use of multiple layers have been known to perform better abstractions for a long time ( see for example An Introduction to Computing with Neural Nets by Lippmann where on figure 14 explains the types of decision regions that can be obtained from adding layers).

2) It allows the creation choke points that forces the neural network to generalize( see Reducing the Dimensionality of Data with Neural Networks by Hinton and Salakhutdinov)

These two reasons may allow you abstract to higher level representations. The problem has been that techniques used on shallow networks such as backpropagation perform badly on such networks, therefore historically limiting research on multiple hidden layers.

There is still much research to be done on deep learning to explain their performance.

This is a really complex question for which I can only give my opinion (I apologize in advance for the long answer).

I will start with a statistical fact that partly explains their success. Deep architecture is currently Glamour and there is an army of great programmers currently using this set of tools on several contests. This correlates with the fact that several contests are won using deep architecture.

Now, let's decompose it into few properties.

1) Even with the L2 regularizer, the classic feed forward neural network does not generalize as well as other methods such as SVM or Random Forests (according to some experiments on 40 datasets using sign test and Poisson binomial test). However, using the drop out or the bagging techniques, you can bring the generalization back to the neural nets and other deep networks techniques (but, I don't have experiments yet to back this claim). Therefore, these techniques should work as well as other methods.

2) The flexibility of their architecture then becomes a tool for incorporating prior knowledge in the problem to solve. Like the kernel of SVM, this can really helps to solve some tasks. However this is a double edged swords that yields many hyper-parameters to tune and became some sort of black art. But now, with the help of Surrogate learning, you can truly speed up hyperparameter learning (see Snoek et al. Practical Bayesian Optimization of Machine Learning Algorithms, 2012)

3) Concerning the "deep" architecture, i think the advantages are two folds. A) it provides a richer class of functions and probably a better prior over it B) Autoencoders provides a great way to take into account the unsupervised data. In some tasks, this unsupervised element is crucial. However, there exists other tools to extract unsupervised representation, such as sparse filtering which won the black box learning contest on kaggle against several deep architectures.

Conclusion : This set of tools can tackle big datasets, incorporate prior knowledge and unsupervised data. So, if you use drop out, fast hyperparameter tuning and other tricks of the trade and run that on a big cluster of computers, you should be able to tackle interesting machine learning problems with great success.

In general, I don't think that the "depth" of a learning algorithm is important. I think it is more a question of the richness of the class of function, the prior you have over it and the amount of data you have for learning. In semi-supervised learning (or domain adaptation, multi-task, transductive... ) you may need an extra layer 1) project into feature space 2) use a basic learning algorithm like SVM or Random Forest (even if it can be seen as a "single" layer).

I totally agree with Alexandre that the class of function, the prior and the amount of data are the most important things to consider. However, I would suggest that using alternate methods while they might achieve the same results, might turn out to be cumbersome. Take for example the traditional backpropagation that it either overshoots or is slow to converge in a solution. Then on top of that using them in cascaded fashion to achieve the same result as a deep learning architecture might not be practical, even though they have been proven to be universal approximators.

Now , with regards to the specific earning tasks such as Speech recognition, my intuition is that it would do better than other algorithms since an analysis of speech might require the following hierarchy: phonological features(10 ms)-> phoneme recoginition((30 ms)-> syllable recognition(128 ms). This stage of the analysis( without even going into words and sentences ) is rather complex and if you cascade the output of one stage to the other the error minimization between the algorithms are difficult to tackle and to troubleshoot. Why not just use a deep belief network to just figure it out and eliminate the hassle (though the hierachy of abstraction might not turn out to be that of . phonological features-> phoneme > syllable, but might be another more optimal abstraction)

Thanks for your references !

I agree with Alexandre when he said that the first fact which explains the excellent results obtained by deep architecture is the number of people working on it.

More deeply (if I can said that), I think that learning the representation by unsupervised training can explain also the excellent results. When I was a PhD student, I have worked on autoassociators for Face Detection (A Fast and Accurate Face Detector based on Neural Networks) . I have used a model which is closed to the one used in Deep Learning. The autoassociator was trained to reconstruct exactly the face images, and was trained to reconstruct as the nearest face image all the other images. A projection on a face set was obtained. The autoassociator used in Deep Learning learn the projection of perturbed inputs to the correct inputs. The set of features obtained on the hidden layer allows to build the inputs and only the inputs. The supervised classification on this kind of features can be more robust.

I think that this kind of models work well.

I would like to point ut some notes that I have mostly read in papers by Dr. Bengio (I will copy my notes here). For more information please refer to [Representation Learning: A Review and New Perspectives]:

1) advantage of depth:

- promote the reuse of features

- progressively more abstract features at higher layers of representations

- more abstract concepts are generally invariant to most local changes of the input

2) local non-parametric learners such as kernel machines mostly only exploit the principle of local generalization i.e. the assumption that the target function (to be learned) is smooth enough so they rely on examples to explicitly map out the wrinkles of the target function. generalization is mostly achieved by a form of local interpolation between neighboring training examples.

3) Kernel machines are useful but they depend on a prior definition of a suitable similarity metric or a feature space in which naive similarity metrics suffice.

4) in previous methods it requires O(N) parameters to distinguish O(N) input regions. In deep learning methods O(2^k) input regions are represented only by O(N) parameters