As success of deep learning depends upon appropriately setting of its parameters to achieve high-quality results. The number of hidden layers and the number of neurons in each layer of a deep machine learning have main influence on the performance of the algorithm. Some manual parameter setting and grid search approaches somewhat ease the users' tasks in setting these important parameters. But, these two techniques can be very time-consuming. I heard alot about the potential of Particle swarm optimization (PSO) to optimize parameter settings.
I want to optimize deep learning parameters to save my valuable computational resources.
The paper in the following link may help you :
https://www.hindawi.com/journals/cin/2015/369298
Actually the deep learning tool you mentioned in your question is MLP.
determining the appropriate values for the number of nodes and hidden layers requires an extensive trial and error process.
Using PSO (or any other evolutionary algorithm ) helps your network to adjust the MLP parameters including weights and biases to achieve a better performance.
A suggestion: be very careful and think how you will evaluate a specific parameter configuration. In very simple terms, PSO will give you sets of real-valued vectors that you need to evaluate and THIS IS THE CRITICAL POINT.
Once you have it clear, you can use genetic algorithms, PSO or wathever carefully designed technique.
Check some papers on training neural networks with metaheuristics first.
Good luck!
David
David A. Pelta I agree with you.You indicated very crucial point.But the question is still as it is.How to adjust deep learning parameters using Particle swarm optimization (PSO) using MATLAB or any simulation software?.Any code which shows simple implementation of Deep learning + PSO to set correct parameters?
Hi:
this is not so simple. As a guideline, I recommend you to use some PSO implementation to optimize a simple real-valued function (Griewank, Rosenbrock,...). Then you will get a feeling on the complexity of what you are trying to do. Most probably, you will need to paralelize the evaluation of every solution (which represents a specific set of parameters).
As the evaluation function will be very costly, it is fundamental to understand the sensitivity of the fitness values with respect to parameters changes in order to avoid making irrelevant evaluation.
Good luck!
David
You CMAES as a greedy to optimize layer by layer then a DE or PSO to optimize the whole
If you are still looking for an answer to this question.
You need to (1) define your fitness function to be the accuracy of the model. (2) initiate a population where each point is a vector containing the number of hidden layers and the number of features per layer (or any other network parameters based on your architecture). For each of these you need to train the model and evaluate the fitness function (calculate accuracy). (3) iterate using position and velocity updates (regular pso) until you satisfy the stopping conditions.
I am not entirely sure how efficient this whole process will be though. (1) You still need to set the parametrs of pso itself, and (2) depending on your network architecture, the time it takes to train, and the parameters of that training, it might be very time consuming, and (3) there is no guarantee of converging to an optimal or near-optimal solution. On the bright side, since the fitness function itself is the accuracy, you can always know if the model is doing better.
Why use a method that does not guarantee success? PSO is only a heuristic - it's not going to provide you with an optimal answer.
You mention the word optimize - and that is something that a metaheuristic in general cannot.
- Badges
- Science topic