I proposed to use a neural network to generalize input-target data from a series of numerical simulations. I intend to use a feed-forward back-propagation neural network and I just need to understand how best to configure the number of hidden layers and amount of neurons in each. I have 6 input values and 6 discrete corresponding target values for each input-target set.
There is an interesting and excellent course on Machine Learning (by Prof. Andrew Ng) at https://www.coursera.org/course/ml. Check the lectures in weeks 4, 5, and 6, I think they contain the information you need.
.......Mostly, check lecture in week 6 (and maybe week 5 the last slides). Week 4 provides just some basic materials of Neural networks
Hi. I'd say, it is impossible to give good concrete answer to your question. The optimal structure and size of your network will depend on many factors.
* How do you normalize the network?
* How do you optimize it?
* How do you initialize its weights?
* What are the relations in your data?
You can either have lot of experience and guess what is a reasonable structure, or you have to search for it (try many different sizes of layers, ...).
In your case, try one and two hidden layers with 4-128 neurons each - run a grid search on a validation set. Do not forget to search for optimal regularization as well.
If you are looking for a sophisticated approach try:
Shoek et al.: PRACTICAL BAYESIAN OPTIMIZATION OF MACHINE LEARNING
ALGORITHMS. 2012.
Hi Alan,
As the above responses suggest, yours is a more complex question than it may seem. It remains unanswerable when framed that way! Neural network architecture is about trade-offs - one of the most well-considered is that between bias and variance. As NNs are data-driven, optimality must be achieved in relationship with the data itself. I have a quote here, "the optimal architecture is a network that is large enough to learn the underlying function and is small enough to generalize well" (Aran, Yildiz & Alpaydin 2009, p. 160). A dataset may reasonably suit more than one structure.
The size of your input and output layers are determined by the data itself, with the number of neurons in the input layer equal to the number of attributes in your dataset and the output layer equal to number of target values (ie 6 and 6).
Increasing the number of hidden neurons, whether they are arranged in a single or in multiple layers) increases the complexity of your model. Early research indicates that increasing the number of hidden neurons improves accuracy in output, (eg, Gorman & Sejnowski 1988) and reduces training time (Denker et al 1987). It has also been demonstrated that this improvement has limits and efficiency is reduced with increasing number of neurons in the hidden layer (Zeng & Yeung 2006).
As your network can not be independent of the data that it is created for, also consider the number of instances used for training as a significant factor in your choice of network architecture. Note here that the sufficiently large training set for a nonparametric classification scheme such as neural networks exists only in theory "since the training data will never "cover" the space of all possible inputs" (Geman, Bienenstock & Doursat 1992, p. 44).
Michal Hradis suggested an exhaustive search with an upper and lower bound on the number of neurons in each of two hidden layers. There has been much consideration on predetermining a network structure. In work on neural networks with a single hidden layer, Barron (1994) and Baum(1988) suggest, among other calculations,
1. #hidden = #instances in set used to train the network / #input neurons.
2. #hidden = #instances in training set / (#input neurons + #output neurons).
As you can see, such calculations are based on a perceived relationship between the training data and the network structure.
If you can source some of the articles referenced here, you may get an idea of the nature of the debate surrounding the question you ask. A short answer is there is no agreement on a correct approach. In the literature you will find many references to "rules of thumb" :)
Amanda
REFERENCES
Aran, Yildiz & Alpaydin 2009, 'An incremental framework based on cross-validation for estimating the architecture of a multilayer perceptron', International Journal of Pattern Recognition and Artificial Intelligence, vol. 23, no. 2, pp 159-190.
Barron, A 1994, 'Approximation and estimation bounds for artificial neural networks', Machine Learning, vol. 14, pp. 115-133.
Baum, EB 1988, 'On the capabilities of multilayer perceptrons', Journal of Complexity, vol. 4, no. 3, pp. 193-215.
Denker, Schwartz, Wittner, Solla, Howard, Jackel & Hopfield 1987, 'Large automatic learning, rule extraction and generalization', Complex Systems, vol. 1, no. 5, pp. 877-922.
Geman, Bienenstock & Doursat 1992, 'Neural networks and the bias/variance dilemma', Neural Computation vol.4 pp. 1-58.
Gorman & Sejnowski 1988, 'Analysis of hidden units in a layered network trained to classify sonar targets', Neural Networks, vol. 1, pp.75-89.
Zeng & Yeung 2006, 'Hidden neuron pruning of multilayer perceptrons using a quantified sensitivity measure', Neurocomputing, vol. 69, no, 7-9, pp. 825-837.
One solution is to use cross-validation and grid search to look the set of parameters that lead to the best results. It's a bit harder with neural nets than it is with some other machine learning algorithms because there are a lot of things to tweak.
A large amount of difference is made by how the input data is pre-processed. Some problems that look difficult and may require a large number of neurons are actually very simple if dimensionality reduction like PCA is used first, then you might find that the data is almost fully described by a few components.
A network with too many nodes for too simple a problem can lead to over-fitting. However, with regularization such as bayesian regularization, drop-out, sparsity constraints, etc this problem can be minimized. With modern neural networks we've figured out how to make use of very large numbers of layers and many neurons. For the older generation simpler feed-forward nets we still have a lot of the same problems as we did when everyone figured out that SVMs were more effective!
At the end of the day it's a lot of trial and error, but grid search might an idea.
The literature posted above is more than enough to get an idea for the modern techniques in neural nets.
- Badges
- Science topic