When we used a deep neural network to training a model for predicting a particular problem, the number of network layer and number of nodes for each layer are the difficult decision in a topic. The trial and error approach is a good method for modifying the number of network layers and number of nodes. The way I often use the grid search approach. Can someone share their own method of setting the number of network layers and the number of nodes in deep neural network?
Dear Yu-Da Lin ,
Following the most recent works, you can learn the neural network architecture (and its hyperparameters). There are many papers working on this issue. You can find more details in:
Article Efficient Neural Architecture Search via Parameters Sharing
Article Progressive Neural Architecture Search
Article Neural Architecture Search with Reinforcement Learning
I believe it would first be prudent to take a step back and understand what level of detail adding layers provides. In every case a set of data has a specific set of assumptions and a specific set of initial inputs which drove the data. Two main points: 1) We like to believe that we can discern all the possible relationships of all the data many times without the initial info, next 2) we like to believe we will find All relationships we didn’t know were there...the truth is even with very small amounts of data it may be just as difficult to get to a positive threshold but outnumber intuition constantly tells us otherwise. Thoughts on solutions: if the choice is any type of ML, just like academic research we must focus on the details from beginning to end. Assumptions are critical, who, what, when, where, how, data collected is also information very important and part of what can help tremendously with decision making on many choices you make as part of ML algo configurations. Lastly, if you approach like a dissertation or true research your outcome algo should be very concise, outliers will have been removed, and algo will match your Data as perfect as humanly possible under whatever ambiguity is left. The main point is that there is not one way to choose. No silver bullet. Some ways work better for some data and not for others. It must be tied to your comprehensive understanding of the situation. i know it’s not the answer you might have been looking for but I think that’s ok. It’s like knowing the exact specifications and ranges for something you build. Hope this helps some.
I agree that the assumptions are important. My approach is to make an estimate on quantity of features to be learned(#features). Count also relationships you want to model as features.
Next step is to focus on quality of the feature separation (how well are features to be distinguishable). Now you have an estimate on a minimal freedom that a network needs in order to store all the features.
1)roughly #features =< #nodes*#feature layers*inter connectivity ratio*#distinctive value intervals per node(how many bits of information can you store per node).
2)apply some dimensionality reduction like PCA to reduce number of features in case NN becomes too big for your HW setup.
3)apply correlation analysis to measure how well features are separated in your set to get some numbers on achievable separation of features
Dear Yu-Da Lin, I used the thumb rules from this paper - Article Huang, G.: Learning Capability and Storage Capacity of Two-H...
It is an important issue specifically in critical applications like medical technologies. My experience was by applying gradient descent in a distributed reasoners to optimize the decision accuracy of my self developed reasoner IK-DCBRC. I estimate this to be worked by changing the error function which need as argument the number of nodes and the measured accuracy. You should define a loop in which the size of ANN will be increased every step after that you measure the error..etc
Good luck.
Dear all experts,
Sorry, I'm late.
This topic is to discuss with experts the experience of the methods used in the past. On the one hand, it can complement and strengthen my model training ability, and it also provides more model training methods through this discussion. Because the data sets collected by different projects are different, if we can learn more model training methods, we can greatly increase the chances of training a practical model, and understand more details that need attention. As we can see, the experts who participated in providing their own experience and inventions and related articles have really improved my horizons. We are honored to receive way to share, but also expect to have more experts to share their experiences together.
If you have some computational time to spare ( which I guess because you use Grid search), a much more focused parameter search can be obtained through Bayesian Parameter Optimization framework (https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f). This is much more effective than grid search and often produces fast results.
HyperOpt (http://jaberg.github.io/hyperopt/) is an excellent python library for Hyper Parameter Optimization. If you are using Keras for your training, you can also check Hyperas (http://maxpumperla.com/hyperas/).
Dear all,
I found the some papers which could facilitate to training the deep learning, including the pre-training method, binary weights, and some interesting strategies.
Best regards
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training