As long as the training and test data doesnt make much significant difference (no noise) and doesnt contain much non-linearity, then those mathematical functional approximations would work well. ANN performs better when there is plenty of noise and the those approximations wouldnt work. Through ANN, we are trying to build a complex model in the form of a network by showing training examples / patterns (involving learning). The network is allowed to further correct the parameters so as to minimize the errors. In the case of mathematical models, in most cases it is a one pass approximation, while the ANN training process is iterative. Most real world data would not work well with mathematical models. Please see link below: http://www.softcomputing.net/ann_chapter.pdf

As long as the training and test data doesnt make much significant difference (no noise) and doesnt contain much non-linearity, then those mathematical functional approximations would work well. ANN performs better when there is plenty of noise and the those approximations wouldnt work. Through ANN, we are trying to build a complex model in the form of a network by showing training examples / patterns (involving learning). The network is allowed to further correct the parameters so as to minimize the errors. In the case of mathematical models, in most cases it is a one pass approximation, while the ANN training process is iterative. Most real world data would not work well with mathematical models. Please see link below: http://www.softcomputing.net/ann_chapter.pdf

The same question applies for any set of functions (i.e. also the set of infinit dimensional vector space SVM is working with). This problem is at the heart of machine learning and there are several different theories for selecting the "right function". The common L2 regularizer comes from theories on linear functions so it may not be appropriate for ANN, but in practice, it works well. The new 'dropout' technique seems to work really well in practice. It does not come from a generalization theory but we could see it as some form of approximation of a majority vote.

Among all these, I strongly recommand bagging of ANNs (or bagging of anything you feel like). If you can't afford to run several big ANNs, I believe that bagging a few small ANNs is better than only having one big ANN. At the heart of bagging lies the idea that since you don't know which of all these functions is the "best" one, you should use the opinion of all functions that seems to work well (i.e. performing well on the training set according to your favorite metric).

I would like to expand on Alexandre's answer answer in that the goal of statistical learning is to approximate the underlying distribution which you do so as you pass more data into the learning algorithm. While the data makes the algorithm approximate the underlying distribution I think that the big advantage of NNs is in the basis function and the activation function. For the case of BP its strength lies in the mixture of the two functions that allows for an input to be noisy and if it passes the activation threshold it will do so. It does not have to be perfect but close enough,and this is its strength. The same argument can be said with radial basis function and other types of NNs.

OK, from a purely theoretical perspective, ANNs are as good as any other classifier, because of the No-Free-Lunch theorem for classifier systems (that states when their performance is averaged over all possible data-sets, all classifiers have the same predictive power...) However, in practice things are clearly different: not all classifiers perform the same, even though it is not usually clear from the beginning which one will have better performance over another.

Finally, regarding the mathematical and statistical aspects of approximation functions, my main source of reference is the excellent book "The nature of statistical learning theory" by V. Vapnik (Springer). It's a very small, concise book that provides a great treaty of the subject, and analyzes the foundations of approximation functions for use in classifier systems. Very powerful theorems are given that help explain the success of ANNs and much more, SVMs. However, the proofs of most theorems are omitted; if you are looking for the proofs also, you should consult either the original papers in which the theorems are proved, or other (more extensive) books by the same (or other) author.

That depends on the training data. I agree with Ajith Abraham. Humans behave in the same way. Image a human that can not distinguish between two gestures because they are similar - machines behave in the same way.

I agree. The nature of statistical learning theory is a great book.

The theoritical framework is beautiful. It gives some clues on how to train a learning machine, in particular it explains what is the objective of training a learning machine: the minimization of the risk.

In a sens, the structural risk minimization answers to the asked question "What makes ANNs good in generalizing some decisions? "

However, the given upper bound on the risk is a function of the VC-dimension h, and we do not know how to evaluate it for a neural network.

There are some other frameworks, which are interesting.

For example if we looks the problem of training an ANN as a problem of interpolation, we understand that training an ANN is an ill-posed problem. Indeed, there are an infinity of functions in the set of functions (the ANN) that can fit all the points. The use of a penality term, such as weight decay, allows to obtain the unicity of the solution.

Then, is this solution a good one ? The Information theory, and in particular the Minimum Description Length framework, can help us. The best solution is the simplier that fit the data. This is an explanation of the use of a regularization term.

The PAC framework (Probably Approximatively Correct) is also interesting. It allows to give practical answers such as : how many training examples for what level of performances ?

From a theoretical perspective, forcing the information through a small hidden layer forces development of a solution that depends on only a relatively small number of features - these features being self-organized on the hidden layer. Using Dimension Reduction techniques like SVD or PCA to reduce the data can also work this way, but it is better not to reduce too much with these techniques, and not to reduce/expand too much in one go in the ANNs...

What do I mean here? Well practically speaking, use multiple hidden layers to map down from a high resolution input matrix (from a medical, visual or auditory source for example) through intermediate features to the minimal representation, and if you have lots of outputs (classes to choose between, or an image type representation as output) step up again gradually on the other side. Generally you won't want to reduce overall to less than 2lgN for N input features, and square roots, cube roots, etc. may be convenient intermediate points. Similarly for the K outputs, you don't want less than 2lgK nodes on the smallest hidden layer, think in terms of bits of information in the most compact representation. This relates to the MDL/MML idea as well as the idea of VC-dimension...

So... if the K outputs are very independent and don't group nicely into related subgroups, the 2lgK might be too small, and you probably want to stick in K^i space for some 0 < i < 1.

Also... In SVMs the margin maximisation explicitly tries to get a simpler solution that maximizes the margin, while for purer Perceptron-based systems or MLPs, early stopping has a similar effect and avoids overfitting. With ensemble, fusion and boosting techniques, there is a danger that you will over fit because you keep trying to fit those errors better, and it is again your validation set that keeps you honest, and ensure you accept only solutions that generalise at least that far.

In relation to the No Free Lunch theorem, it can be a positive rather than a negative... you just need to select your biases carefully. So for example, if dealing with sensor data related to the real world, we use biases related to Euclidean distance, the idea that objects and events tend to stick together in space and time - my arms and legs tend to stay pretty close and I don't often leave my head behind, and my ability to act at a distance is pretty limited really! In fact, Cognitive Linguistics theorizes that that is why language is the way it is with biases about spatiotemporal nearness in the world, apply also to nearness of other features, and similarity of features we use for communication, so these biases are appropriate for speech, gesture and music too.

With the right biases, you are more likely to get a small compact understandable and effective theory or model - connecting back to Parsimony and Entropy and MDL and MML.