Hi,

There is a long list of optimization algorithms that you can use to tune the hyperparameters of machine learning models. Regarding metaheuristic optimization, a genetic algorithm has been implemented in the library sklearn-genetic-opt to optimize hyperparameters, such as the number of trees, learning rate, and depth of the tree. The library scikit-opt implements other metaheuristic algorithms, such as particle swarm optimization and simulated annealing.

All these hyperparameter strategies can be used to tune gradient boosting models (or any machine learning model). In this sense, you can use these strategies to compute a set of hyperparameters that maximize the model performance.

Other optimization algorithms are widely applied in hyperparameter optimization tasks, for instance, Bayesian optimization (python libraries: scikit-optimize and hyperopt).

Thank you for your answer

Hello Kumudu,

The metaheuristic algorithms can be used to train the model and tune hyper parameters. The challenge with this algorithm is that it can give you different results with different runs as they are stochastics algorithms.

The metaheuristic algorithm will be used to minimize a function that can be the cost function generally. and you can go through that

It is possible and you can find several papers that implement this strategy.

The first step is identifying the catboost internal parameters and set a lower and upper bounds for them (in order to define a search space).

Then is necessary build a fitness function to be optimized by the evolutionary algorithm. This function must involve the catboost internal parameters.

There are some evolutionary computation libraries that can integrate catboost and an evolutionary algorithm easily.

You can send me an email, to send you the python code of catboost linked with INFO optimization algorithm.

For more detail about INFO algorithm you see my website.

imanahmadianfar.com

I was able to develop the code. Thank you for all of your answers.