Hi, I am doing a fitting problem. The method I've finished is least square fitting, which doesn't look good. I am now considering gradient descent. Could anyone provide some insights about the benefit of gradient descent? What's the major advantages of gradient descent compared to least square?
Least squares is a special case of an optimization problem. The objective function is the sum of the squared distances.The solution can be found analytically.
Gradient descent is an algorithm to construct the solution of an optimization problem approximately. The benefit is that it can be applied to any objective function, not just squared distances.
So if you find the least square result unconvincing, squared distances are maybe not the adequate objective function for your problem. If you find an alternative one, gradient descent is a way to obtain at least a locally optimal solution.
I recommend these tutorials
https://towardsdatascience.com/linear-regression-simplified-ordinary-least-square-vs-gradient-descent-48145de2cf76
https://www.youtube.com/watch?v=IyDwQNXDWns
https://www.youtube.com/watch?v=wjhT2YFNmlw
LS is one of the method of data smoothing. You also can use interpolation, splain, etc. approaches. To find optimal point, you also can use different methods. One of them is gradient descent. But... If you task (e.g., LS) have a lot of local minimums, gradient descent works bad. In this case you should use other algorithms - genetic, cross-entropy, tabu, etc.
Wenbin Wang Gradient descent basically find optimal parameters in small step. Moreover, it's more general approach you can apply to any optimization problem but it's slow as it optimizes in small steps. Whereas Least Square deals with specific type of optimization i.e. Sum of Square. Coming to key point it depends on which type of data you are fitting as best to my knowledge as highlighted by others also gradient descent can be a good choice as it's general.
Thomas Neusius Cristian Ramos-Vera Sergey Porotsky Siraj Munir Thank you, all.
As others already mentioned the gradient descent is not an alternative for the least-square method. You have an error functional (the difference between you regression hyper-plane and your data) and you want to find the best parameters to minimize it. So, either you solve this with normal equations (using the first-order optimality condition) or QR decomposition (which simplifies the error representation) or an iterative method to minimize the error directly (like what the gradient descent method is doing). Note that the other methods give you the best possible result while, as the GD method is iterative, you should be careful in choosing the step size to converge to the correct solution.
In general, if you do not get the right solution, you may want to change your fitting function. Perhaps it will help.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics