The technique of changing the optimizer during training steps is commonly referred to as "learning rate scheduling" or "adaptive learning rate." Instead of using a fixed learning rate throughout the entire training process, learning rate scheduling involves adjusting the learning rate dynamically based on certain criteria or predefined schedules.

There are several variations of learning rate scheduling, and one of the approaches involves changing the optimizer rather than just the learning rate. An example of this is the technique known as "optimizer scheduling" or "adaptive optimizer." In this approach, different optimizers are used at different stages of training to benefit from their respective strengths.

For instance, a common strategy is to start training with a more computationally efficient optimizer like Stochastic Gradient Descent (SGD) and then switch to a more sophisticated optimizer like Adam or Adagrad as the training progresses. This helps to balance the benefits of faster convergence in the initial stages with the more refined optimization capabilities of certain algorithms in later stages.

The key idea is to adaptively adjust the optimization strategy during training to overcome challenges such as oscillations, slow convergence, or overshooting that may occur with a fixed optimizer. This adaptive adjustment can lead to improved model training and convergence.

In general we call it hyper parameters tunning. And if you are investigating the effect of each layer or path in the network then it's ablation study. For more details you can go through this Article Multi-horizon short-term load forecasting using hybrid of LS...

I consulted with ChatGPT, and it gave a nice answer “Optimizer switching” or “Optimizer annealing”. I actually like the term “Optimizer annealing” as the idea behind it was to imitate simulated annealing method.

While I do agree that optimizer scheduling sounds close to the truth, I do agree that it sounds more like parameters tuning instead of actual optimizers changing. To be clear about it, I will quote ChatGPT reply:

"While there is some overlap in the concepts, "optimizer switching" and "optimizer scheduling" are not exactly the same, though they share the common goal of improving the optimization process during the training of machine learning models.

While optimizer scheduling often involves adjusting the learning rate, it doesn't necessarily involve changing the entire optimizer itself. In contrast, optimizer switching specifically focuses on changing the optimizer, which may include a change in the optimization algorithm and associated hyperparameters.

In practice, researchers and practitioners might use a combination of optimizer scheduling and optimizer switching to achieve the best performance for a given task. The choice between these techniques depends on the characteristics of the optimization problem and the behavior of the model during training."

The process of changing the optimizer during the training steps of an Artificial Neural Network (ANN) is generally referred to as "optimizer scheduling" or "learning rate scheduling." This technique involves adjusting the learning rate or the entire optimizer configuration over the course of training to enhance the model's convergence and performance. Instead of using a fixed learning rate, optimizer scheduling allows for dynamic changes based on the progression of training epochs. Common strategies include reducing the learning rate exponentially, step-wise, or using more complex schedules like cyclical learning rates. This approach helps address challenges such as overshooting or slow convergence, offering a dynamic adaptation to the evolving characteristics of the optimization landscape. While it falls under the broader umbrella of hyperparameter tuning, optimizer scheduling specifically focuses on the optimization algorithm and its parameters during the training process.

Deep learning.

For example: Gradient Descent can be considered the popular kid among the class of optimizers in deep learning. This optimization algorithm uses calculus to consistently modify the values and achieve the local minimum

In Machine alearning event, this week, I had a conversation with some leading scientists and reply was that it can be called Optimizers Switching or I can just name it in different name in research paper. I think I will stick with Optimizer Switching (OS).