How is the depth and width of a neural network in machine learning chosen for optimal processing?
Given that neural networks are tending to be deeper now (thanks to more efficient coding and faster machines) there's greater variability in the number of parameters to choose from. However, in spite of this 'spoilt for choice' scenario, there must be general guidelines on the number of layers and the widths of these layers in a neural network, so networks can be trained more rapidly, data is evaluated more quickly and computer resources are not wasted.
Input data parameters are normally fixed, such as the number of input nodes and the dynamic range in the data. Output parameters are also fixed, such as the number of output nodes and the quality of the data. Furthermore, the complexity of the neural network goes linearly with the depth of the network, but as the square of the width of the network.
Since the above 'boundary conditions' are generally known for an application of neural networks, how does one go about choosing the width and depth of a neural network?
many thanks,
neil
I would like to add to Aparna Sathya Murthy 's answer and link to this paper about Efficient Nets: Preprint EfficientNet: Rethinking Model Scaling for Convolutional Neu...
The researchers working on the project seemed to have found a sweet spot between the trade-off of width and depth. Cheers, Raoul
In a Neural Network, the depth is its number of layers including output layer but not input layer. The width is the maximum number of nodes in a layer. If want to know furthermore, you can go through textbook written by me on "Fuzzy logic and Neural network.
Are you into regression or classification problems or anything else? For time series problems, there are information criteria such as the Murata et.al. (1994)'s MIC designed for NN's. But this was for sigle layered NN's and you should estimate a number of models to differentiate between them. There are also statistical methods such as F tests. Both methods require serial computing. So, computational power in addition to the complexity of the data and the problem are very important. All below are for time series models:
I suggest an extensive data mining to predetermine some idea about the input layer first and work on it hard. This is especially important to gain time to overcome the PC's abilities :) I suggest a linear regression problem first, then a polynomial regression to work on the determination of the input layer without going very deep. Then, the selection of model and most importantly the learning algorithm are very important. Different codes also produce different results, try the most cited ones. For single layer NN's, I suggest estimating a large number of NN's, like 200 randomly selected models with varying no of inputs, in addition to no. of neurons ranging between 2-10 for simplicity for my time series data depending on the time series problem. A similar method could work for you. I also suggest algorithm coorporation + weight decay + early stopping. Select the models with maximum predictive accuracy. This works for me for MLP's, single layer recurrent nets and jump connection NN's but could work well if computational complexity is not a burden. For deep NN's, the importance of learning algorihm becomes especially relevant. For DNN's designed for time series such as the LSTM, I still use the first method for determining the input layer at the first step. Depending on the type of the problem, evaluate the codes and softwares very carefully.
p.s. I suggest deep NN books as suggested. However, you should also try the fun part: Have you tried the Kaggle competitions and blogs? I strongly recommend in addition to academic books and articles.
- Badges
- Science topic