Setting initial parameters is quite important in many ML and specifically, in deep neural networks. Weights and biases are the parameters of a deep neural Network. So what is the best weight initialization strategy in Deep Convolutional Neural Notworks?
In this regards, is the unsupervised pre-training of a deep NN a better strategy to set initial weights at good locations in weight space for the supervised learning phase.
Dear Asifullah,
Please read the following papers:
1. Xu, L., Ren, J. S., Liu, C., & Jia, J. (2014). Deep convolutional neural network for image deconvolution. In Advances in Neural Information Processing Systems (pp. 1790-1798).
2. Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015). Training very deep networks. In Advances in neural information processing systems (pp. 2377-2385).
3. Larochelle, H., Bengio, Y., Louradour, J., & Lamblin, P. (2009). Exploring strategies for training deep neural networks. Journal of machine learning research, 10(Jan), 1-40.
Thanks,
Sobhan
Dear Asifullah,
Please read the following papers:
1. Xu, L., Ren, J. S., Liu, C., & Jia, J. (2014). Deep convolutional neural network for image deconvolution. In Advances in Neural Information Processing Systems (pp. 1790-1798).
2. Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015). Training very deep networks. In Advances in neural information processing systems (pp. 2377-2385).
3. Larochelle, H., Bengio, Y., Louradour, J., & Lamblin, P. (2009). Exploring strategies for training deep neural networks. Journal of machine learning research, 10(Jan), 1-40.
Thanks,
Sobhan
For any Neural Network, as per the literature of course, the weight should follow normally distribution initialization and should be random values between 0 and 1. DNN could be convolution, lstm or any other type.
See
https://stackoverflow.com/questions/20027598/why-should-weights-of-neural-networks-be-initialized-to-random-numbers
https://stats.stackexchange.com/questions/47590/what-are-good-initial-weights-in-a-neural-network
https://medium.com/@amarbudhiraja/towards-weight-initialization-in-deep-neural-networks-908d3d9f1e02?source=linkShare-f1aea8ad02fd-1519979357
Dear Asifullah
please follow the links
https://stats.stackexchange.com/questions/47590/what-are-good-initial-weights-in-a-neural-network
https://arxiv.org/pdf/1512.07108.pdf
https://pdfs.semanticscholar.org/fc5b/db98ff97581d7c1e5eb2d24d3f10714aa192.pdf
https://arxiv.org/pdf/1511.06856.pdf
- Badges
- Science topic