Up To what resolution can non-linear optimization problem algorithms work and solve nonlinear optimization problems?
I have a simulation as test case, so i know the global solution, and the coefficient values thereof.
So, theoretically, the optimization is 0.
Yet, What would be the practical residual value the optimization algorithm will and can yield? Can it be lower than 1e-15, or even lower? What about 1e-20, or so?
I am already working with initial values with residual of 1e-9.
For numerical computations, roundoff error is a fundamental limit; for IEEE double precision arithmetic, unit roundoff is about 2.2e-16, so unless you are dealing with very small or very large numbers, you are unlikely to get better than about 1e-15 within the optimal. Of course there may be other issues that make it hard to get even this far. Gradient descent algorithms tend to be slow and this means that the results are usually of (relatively) low accuracy. (e.g., getting within 1e-6 of optimal with gradient descent is usually pretty good for that method).
To say much more, I would have to know a lot more about the problem you are solving and the method you are using.
For numerical computations, roundoff error is a fundamental limit; for IEEE double precision arithmetic, unit roundoff is about 2.2e-16, so unless you are dealing with very small or very large numbers, you are unlikely to get better than about 1e-15 within the optimal. Of course there may be other issues that make it hard to get even this far. Gradient descent algorithms tend to be slow and this means that the results are usually of (relatively) low accuracy. (e.g., getting within 1e-6 of optimal with gradient descent is usually pretty good for that method).
To say much more, I would have to know a lot more about the problem you are solving and the method you are using.
In principle optimization algorithms, if correct (means, there is a proof of convergence to a (mostly only local) solution), can solve it to any required
precision provided your computer arithmetic provides the means for this.
There exists arbitrary precision arithmetic in the net, hence this is a question
of effort only. But since real life optimization problems depend on modelling
real life objects by mathematical objects, hence
("all models are wrong, but some models are useful", citation from Williams)
and this makes it questionable whether this effort makes sense
The usual floating point arithmetic of modern CPUs that these numerical solvers rely on works only to a certain precision. As mentioned above, a precision of 2.2e-16 is the best you can expect. However, most solvers need a certain portion of this theoretical precision for themselves, to make sure that their internal calculations are correct. So typically, the numerical solvers offer a precision between 1e-6 and 1e-9 to the user, and use the remaining digits (up to 1e-16) for internal purposes.
- Badges
- Science topic
- Similar topics
- Mathematics
- Applied Mathematics