Hi Bahaa El-Din Helmy

Applying meta-heuristics to any concept for hyperparameter tuning is just like solving for an optimization problem. The only difference is that the objective function becomes the output of the algorithm you are trying to tune.

For example, say you have a neural network, and you need to fine tune the number of neurons and the number of layers within the neural network. First, you would model a chromosome such that it contains values for each parameter you are tuning. For this example, your chromosome would look something like this:

Chromosome = [number_of_neurons, number_of_layers]

Thereafter, you would define your objective function as the neural network, and you would take the output of the neural network (lets say the output is the accuracy) and evaluate its performance by passing through the chromosome values into the objective function.

After those two steps, everything else is the same as coding a meta-heuristic algorithm. What i mean is, the steps thereafter will remain the same:

• Crossover
• Mutation
• Evaluation
• Termination

I have attached the code for my application using meta-heuristics to fine tune the hyperparameters of a GA that I had designed.

I hope this helps.

Good luck!

You can see this paper:

Article Learnheuristics: Hybridizing metaheuristics with machine lea...

For any machine learning and deep learning problems, you have some hyper parameters which should be tuned in order to obtain the suitable results.

These hyper parameters should be optimized. You can also predict these parameters with trial and error. But metaheuristic algorithm are more accurate.

you can define a metaheuristics optimization, such that the hyper parameters are the input of objective function and the neural network performance is objective function.

For example for a neural network, you can define the number of hidden layer and node as a input of optimization problem, then you can optimize this parameters such that the performance of neural network increases, and obtain the optimal value of parameters utilizing metaheuristic optimization.

I hope I was able to get what I meant. :-)

@Mohsen Rezaei .. Thanks for your reply..

Do you have source code?

I totally agree with MR. Rezaee and consider these links:

Research Hyperparameter Tuning

Code MACHINE LEARNING - HYPERPARAMETER TUNING

good luck

All neural networks don't have direct mathematical solution.

You can use tensor-flow or pytorch for optimization of hyper parameter for practical purpose !!!

Use autokeras.

Bayesian optimization and its variants are the state of the art for hyperparameter optimization. There are a wide range of libraries you can use in Python like Optuna and Hyperopt. Regards.

What is the type of your data and What is its size and what are the capabilities of your computer

Metaheuristic was used to adapt hyperparameter of Deep Learning, but what is the training data size and what is your computer capabilities.

The first thing is the dimension optimization means no. of parameters required to be tuned. Define objective function as per requirement.

Collect the data and import on matlab for training then extract the best trained model.

Could you tell me ur deep learning topology which u r using.

Means Supervised or unsupervised learning.

Deep Learning is a vast field of research and metaheuristics vary based on the group of the algorithm i.e. for example some metaheuristics are graph search-like algorithms, some are constructive, another type is combinatorial, etc.

I highly believe that metaheuristics can be used for features selection at the last layer of your network, to guide the dropout within the network instead of keeping it random, and another approach could be changing the topology of the network or the connection of hidden layers based on some assumptions from an epoch to another. Yet, the last option is computationaly expensive. You can also use the metaheuristics to test different scenarios of hyperparameters i.e. constructing the best set of hyperparameters from predetermined values for each hyperparameter you believe you need to change.

P.S. You cannot take for granted that each metaheuristic can be applid to any problem because the conventional process would be:

1. Determine the problematic

2. Decide on what should be optimized when opting for deep learning

3. Model the solution based on a suitable metaheuristic

4. Run experiments

5. Record findings

6. Report best findings and worst cases alike

It is also paradox to say that most metaheuristic algorithms tend to have emperic values i.e. as if you have meta hyper-parameters but still they were successfully merged with deep learning in a lot of architectures.

I totally recommend that you read this review:

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review by Mehrdad Kaveh and Mohammad Saadi Mesgari.

For collaboration I suggest that you reach out to https://www.researchgate.net/profile/Nabil-Neggaz

and for enqueries you can check with

https://www.researchgate.net/profile/Seyedali-Mirjalili