I came across a picture denoting neural network that has more nodes (variables) in an hidden layer than the input layer. I do not know why it makes sense to have more variables than the input as an intermediate output. Can somebody please help me? Thanks in advance!
Hello Joy, I think it depends on the number and complexity of relevant features.
As a simple and stupid example, imagine an input layer organized as 4 x 4 matrix = 16 nodes. For some reason, the nodes in the hidden layer should detect patterns in the input layer containing straight lines. If we neglect learning algorithms for the moment, and design the hidden layer and its connections and weights manually, a reasonable approach were to assign one node to each possible straight line. Now there can be 4 horizontal, 4 vertical, and 10 diagonal lines, resulting in 18 nodes in the hidden layer.
Apart from the fact mentioned above, that the size of the layers represents the network complexity, there is also an aspect of representation quality.
In a few papers higher-dimensional representation were used to make the learning problem easier. Can provide you some references if you are interested.
For example, one can use Sparse Autoencoder with middle layer having more nodes than input later to achieve sparse representation: this representation would contain more numbers, but most of them would be zeros. It is computationally efficient and great for interpretability.
There is also the assumption that learning results in only a small fraction of the weights being used, which were close to the solution in the first place. The probability of your initial weights containing helpful parts increases with the number of weights. So over-parameterization does not have to be a bad thing.
Joerg Fricke , Taras Kucherenko , and Clemens-Alexander Brust thanks for sharing your thoughts toward the question. With Joerg's concrete illustration, I now know that there are cases when having more nodes in the hidden layer is necessary. And with Taras' answer, I get a sense of when having more nodes in the hidden layer can be helpful. And with Clemens-Alexander's answer, I understand that having more nodes in the hidden layer does not mean the model is bad. All three answers are helpful, you have my gratitude!
I agree with Danfeng Hong
in most cases, we are trying to learn patterns that are not obvious in the input layer. As such, we have to extend the inputs to a hidden layer with more nodes where the patterns may be now be identified.
Danfeng Hong
and Abubakar Bala , thanks for the answer. Actually, I just saw an case where the original inputs are three binary variables and the hidden layer are 8 classes that denote the and/or/not relation among the three binary variables. I guess this is an example that matches your answer.
If one is trying to implement a logic gate in your NN, such as an exclusive disjuntion of two attributes, you would of necessity need to have more nodes in the intermediate layer.
Abubakar Bala this is a very good question. In fact, there is no exact rule to set the size of the hidden layer. For some networks, setting the hidden layer's size bigger than the input one is a recommendation such is the case of Echo State Network (you can see my list of publications and pre-pints). However in this case, it is recommended also to use a sparse hidden weight matrix in order to ensure the stability and decrease the comlexity of the hidden layer. I think that the sparsity rate of the hidden weight matrix is a main constraint to pay attention to when setting the hidden size higher than the inputs number
Need to cast the neural "net" wide and then prune later.
Sorry for all the metaphors!
- Badges
- Science topic