Dear the experts,
I am looking for the way to treat model complexity in terms of input/ output dimension or the size of domain/range of it.
For me, it is trivial that the model complexity is reduced when the output dimension changes from 10 to 1, for example.
And, for the similar situation, it would hold for the case when output range becomes to the discrete, such as [0,1], from continuous R.
I wonder if it is able to be identified by a sort of quantitative approach.
If there is some related papers, or related ideas, it would be really helpful for solving the question.
Thank you for the attention.
Hi,
Generally speaking, there can be two ways to determine the model complexity.
1) Bayesian information criterion (BIC): This is all about what you are after, you can determine the number of trainable/free parameters in the model.
2) Akaike information criterion(AIC): This is more about model quality.
I would provide two examples to show how to determine the number of trainable parameters in the deep learning model.
1) Fully connected DNN:
Parameters in a layer (n) = outputs x (inputs + 1)
So you can compute the number of parameters in all layers and get the summation.
2) RNN-LSTM:
Parameters in a layer (n) = 4 x outputs x (inputs + outputs + 1)
Best,
Kadupitiya.
Hi,
Generally speaking, there can be two ways to determine the model complexity.
1) Bayesian information criterion (BIC): This is all about what you are after, you can determine the number of trainable/free parameters in the model.
2) Akaike information criterion(AIC): This is more about model quality.
I would provide two examples to show how to determine the number of trainable parameters in the deep learning model.
1) Fully connected DNN:
Parameters in a layer (n) = outputs x (inputs + 1)
So you can compute the number of parameters in all layers and get the summation.
2) RNN-LSTM:
Parameters in a layer (n) = 4 x outputs x (inputs + outputs + 1)
Best,
Kadupitiya.
.
the following might be of interest :
https://arxiv.org/pdf/1802.07044.pdf
The Description Length of Deep Learning Models
Léonard Blier and Yann Ollivier
NIPS 2018
Solomonoff's general theory of inference and the Minimum Description Length principle formalize Occam's razor, and hold that a good model of data is a model that is good at losslessly compressing the data, including the cost of describing the model itself. Deep neural networks might seem to go against this principle given the large number of parameters to be encoded. We demonstrate experimentally the ability of deep neural networks to compress the training data even when accounting for parameter encoding. The compression viewpoint originally motivated the use of variational methods in neural networks. Unexpectedly, we found that these variational methods provide surprisingly poor compression bounds, despite being explicitly built to minimize such bounds. This might explain the relatively poor practical performance of variational methods in deep learning. On the other hand, simple incremental encoding methods yield excellent compression values on deep networks, vindicating Solomonoff's approach.
.
- Badges
- Science topic