Steftcho P. Dokov
getting trapped in one of the local extrema always causes troubles during optimization
0 votes
0 thanks
Farid Leguebedj
Local minima / Saddle points — Gradient descent can get stuck in non-optimal points. Vanishing / Exploding gradients — Especially in deep networks, gradients can become too small or too large to train effectively. Poor scalability — Struggles with massive datasets or very high-dimensional parameter spaces. Sensitivity to hyperparameters — Requires careful tuning of learning rates and other settings. Non-convex optimization — Modern AI models often have highly non-convex loss surfaces, making convergence hard and slow. Advanced optimizers — Use Adam, RMSProp, or adaptive methods to improve stability and speed. Better architectures — Techniques like residual networks (ResNets) and normalization layers mitigate gradient issues. Regularization and initialization — Dropout, batch norm, and smart weight initialization help guide optimization. Second-order methods / Quasi-Newton — Though computationally heavier, they can escape bad local regions. Meta-learning and AutoML — Automate hyperparameter tuning and optimizer choice. Stochastic methods & heuristics — Stochastic gradient descent (SGD), momentum, and learning rate schedules boost performance on large datasets.
Current limitations of traditional techniques (like gradient descent):
How can they be overcome?
0 votes
0 thanks
- Badges
- Science topic
More Bensafi Toufik's questions See All
Similar questions and discussions