The conventional way of parameter estimation is Maximum Likelihood Estimation (MLE). The second is Expectation Maximization (EM) methods or Variation methods. You can also use SVM or Regression methods, however., all this model selection is based on you choice of data, i.e., weather you considering text data or images data etc.,?

Please, see article "The Cross-Entropy Method for Continuous Multi-Extremal Optimization" on my RG page

Article The Cross-Entropy Method for Continuous Multi-Extremal Optimization

Dear Usha Nandhini ,

In the field of machine learning, hyperparameter optimization refers to the search for optimal hyperparameters. A hyperparameter is a parameter that is used to control the training algorithm and whose value, unlike other parameters, must be set before the model is actually trained.

Here are six techniques to boost your machine learning models:

1. Grid Search

Grid Search is a traditional way to search for optimal hyperparameters. An exhaustive search is performed on a manually defined subset of the hyperparameter space of the learning algorithm. A grid search must be guided by a performance metric that is typically calculated by cross-validating on training data or validation data not considered during training.

For real or unlimited spaces of individual hyperparameters, a discretization and limitation to a part of the space must be defined before the grid search. (More Info: Grid Searching in Machi

ne Learning )

2. Random Search

In random searches, instead of trying out all combinations exhaustively, a random selection of values is made within the given hyperparameter space. In contrast to a grid search, no discretization of the space is necessary. The random search can exceed the grid search in speed and performance, especially if only a small number of hyperparameters influence the quality of the learning algorithm. The reason for this is that the grid search only tries out a few values for each parameter (but several times), while randomly selected values are much better distributed in the search space. (More info: Wikipedia)

3. Bayesian Optimization

Bayesian optimization is a global optimization method for noisy black-box functions. It uses validation data to build a probabilistic model of the function between hyperparameter values and the metric to be evaluated for hyperparameter optimization. In the process, iterative hyperparameter configurations are tested that appear promising according to the current model, and the model is then adapted using the new findings. Bayesian optimization tries to collect as many observations as possible about the function, especially about the position of the optimum. At the same time, it takes into account the exploration of areas where there is little knowledge about the expected performance (exploration). The exploitation of knowledge about areas where the optimum is expected (exploitation). In practice, it has been shown that better results can be achieved with Bayesian optimization due to the collected knowledge than with grid or random search in fewer experiments. (More info: Wikipedia)

4. Gradient-based optimization

For some learning algorithms, it is possible to calculate the gradient in relation to the hyperparameters and to optimize them by the steepest descent method. The first application of such techniques took place for neural networks. Later they were also used for other models such as support vector machines and logistic regression.

Another approach to obtaining gradients in relation to hyperparameters is to differentiate the steps of an iterative optimization algorithm automatically.

5. Evolutionary optimization

In evolutionary optimization, evolutionary algorithms are used to search for the global optimum of a noisy black-box function. The evolutionary hyperparameter optimization follows a process inspired by evolution:

Evolutionary optimization has been used for hyperparameter optimization for statistical learning algorithms, automated machine learning, and the search for architectures of deep neural networks. (More info: Wikipedia)

6. Population-based

Population-Based Training (PBT) learns both hyperparameter values and network weights. Multiple learning processes operate independently, using different hyperparameters. Poorly performing models are iteratively replaced with models that adopt modified hyperparameter values from a better performer. The modification allows the hyperparameters to evolve and eliminates the need for manual hyper tuning. The process makes no assumptions regarding model architecture, loss functions, or training procedures. (More info: Population-Based Methods )

References :

6 Techniques to Boost your Machine Learning Models

https://www.aisoma.de/6-techniques-to-boost-your-machine-learning-models/

From Basic Machine Learning to Deep Learning in 5 Minutes

https://www.deepnetts.com/blog/from-basic-machine-learning-to-deep-learning-in-5-minutes.html

Top Optimisation Methods In Machine Learning

https://analyticsindiamag.com/optimisation-machine-learning-methods-gradient-descent/

this link may be help you

https://www.mathworks.com/matlabcentral/fileexchange/71120-optimal-feature-selection-for-knn-classifier

Dear Usha Nandhini The harmony search algorithm is among the optimization algorithms that you can use it to optimize the parameters.