I want to train a nonlinear system with MLP neural network in PyTorch , how can found the optimum value of hyper parameters such as the number of hidden layers and nodes?
Dear Mohsen Rezaei ,
you can use hyperparameters search methods such as grid search.
Best,
Well, no specific answer in this regard; you can select a range of layers (number of layers), and got your results. Then you can use the ROC curve against the number of layers and the effectiveness achieved. An example of the ROC curve used is here (Article Class Distribution-Aware Adaptive Margins and Cluster Embedd...
)
ِِDear Amin Honarmandi Shandiz , I am completely familiar with GridSearchCV, But the computational cost is increased due to a large amount of data, I am seeking for a straightforward way method that predict the number of hidden layer and nodes without any optimization, for example a simple formula or something like that.
Generate the ROC curve against the number of layers and the accuracy obtained.
Hi dear Mohsen Rezaei. I completely agree with Khurram Hameed. Other methods do not work, since each task has its own specifics. I tried all the formulas for calculating the number of layers and neurons, and compared them with the experimental results, they did not agree. Finding the number of layers and neurons in layers is creativity, without a series of computational experiments it will not work.
In my projects, I do the following: I make up the initial (draft model) at the beginning, for example, I select the number of neurons of the first layer at least the number of input data elements. On the output layer, I select the number of neurons based on the number of output signals. I choose the type of optimizer, the type of Loss function, and the metric for evaluating the quality of the network. And then I run the model and start optimizing the number of layers and neurons in the layers sequentially. And at the very end, I conduct a series of experiments to find the best type of optimizer.
Hi Mohsen Rezaei,
There is no one solution that works for all problems. It depends on your problem. More hidden layers and neurons mean more parameters, construction of features at different levels of resolution and a more complex representation. You can start with quick implementation, try some range of values to get intuition in which direction you need to go, i.e. to increase or decrease the complexity of your model. Plotting learning curves can help with this.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training