I have a Belief Network (Stochastic Neural Network) which is trained using Wake-Sleep algorithm (as defined in the attached paper). What is the best method to avoid overfitting of this model?
Especially, I am concerned about choosing number of the units (stochastic neurons) in each layer. How to choose an appropriate number of units, in terms of good generalization?
Note that my problem is totally unsupervised, i.e. there are no class labels for the data. Actually, I want to use the Belief Network as a generative model in a kind of "synthesis" problem (not classification).
http://search.proquest.com/openview/46b0529a5ebd4e1add1aebdb5bb74a31/1?pq-origsite=gscholar
There's loads of possible reasons. Hard to tell without seeing the code or data. Do you have the correct hyper-parameters (you can use grid or random search to choose), are you using some sort of cross validation, how are you initiallising the parameters. All of these things effect how the network learns. Also possible that your code has a bug!
.
i would keep a validation set (or use a cross-validation) and estimate the upper bound of the description length (this upper bound is just the quantity which is being minimized by the WS algorithm) for different choices of neural network structures (number of layers, number of units per layer, type of units ...), then pick the structure with the lowest upper bound
(recall that minimizing the description length is indeed a right way to fight overfitting, this is indeed the rationale behind the WS algorithm)
see
https://papers.nips.cc/paper/1153-does-the-wake-sleep-algorithm-produce-good-density-estimators.pdf
of course, this may be quite costly in terms of trainings if you want to test a lot of different structures ...
.
.
oh ... and if you plan to go along the "deep" way, you may have a look at
https://arxiv.org/abs/1406.2751
which does not seem to support the choice of sigmoids
(but that is for deep networks)
.
- Badges
- Science topic