Hello! For Global Optimization I often use random-search oriented methods, such as Genetic Alg., Cross-Entropy Alg., etc. I always see, that accuracy and speed of convergence very large depend of Goal Function structure and parameters. For example, for MLE optimization we can use Weibul CDF as
{1-exp(-a*t^b)} or {1-exp(-t/w)^b)}. May be, you have read some generic recommendations, how to select structure and set of Goal Function parameters.
So, you are asking what parametrization of the otherwise well defined Goal Function (better known under the name of objective function) is better in terms of how quickly you will arive at final solution. In the example shown the choice is between the pair (a,b) and (w,b). They are mathematically equivalent, but in computational practice one of them may be better than the other. Being a physicist I prefer the second form in which it is clear that b is dimensionless while w is expressed in the same units as t. The other reason for my preference is that it is probably easier to estimate initial values for w and b than for (a,b) pair. Needless to say that good initial guess never harms.
In more general setting: the objective function should be close to a quadratic form of unknown parametrs at least when considered near optimum (or equivalently: its gradient should be close to multilinear expression in unknowns). This, however, is easy to check when you already know the solution
Dear Sergey:
what do you mean by Goal Function? If tyou are talking about the objective or the cost function, then this is defined by the problem at hand and you have to deal with.
If you could clarify your question, then you may obtain some help.
Regards
David
Dear David, Hello and thanks for your answer. For Goal Function I mean, as usually, the function, which we should to minimize. About "then this is defined by the problem at hand and you have to deal with". It seems me, it isn't right. For example, for search of the CDF parameters by means of MLE approach we can use for optimization initial Likelihood or NLL (Negative Logarifm Likelihood), can use Weibull CDF on the form with parameters Alpha and Better or on the form with Teta and Better, etc. The question is how to select appropriate form of the Goal Function - certainly, from optimization point of view!
Dear Sergey:
the cost function essentially measures the quality of a potential solution. At the end, is the user responsability to design the mathematical function that best reflects his idea of "quality". If there is more than one alternative, I would suggest a multi-objective approach.
Otherwise, you may end optimizing something irrelevant because of the use of a "nice" cost function. (for example, trying to design a differentiable function). I doubt that a general methodology exists.
Regards
David
So, you are asking what parametrization of the otherwise well defined Goal Function (better known under the name of objective function) is better in terms of how quickly you will arive at final solution. In the example shown the choice is between the pair (a,b) and (w,b). They are mathematically equivalent, but in computational practice one of them may be better than the other. Being a physicist I prefer the second form in which it is clear that b is dimensionless while w is expressed in the same units as t. The other reason for my preference is that it is probably easier to estimate initial values for w and b than for (a,b) pair. Needless to say that good initial guess never harms.
In more general setting: the objective function should be close to a quadratic form of unknown parametrs at least when considered near optimum (or equivalently: its gradient should be close to multilinear expression in unknowns). This, however, is easy to check when you already know the solution
Dear Sergey, I answered your question just because I encounter similar problems in my interval calculations. When a and b are intervals (ranges of numbers) then instead of evaluation of a-ab+b you better use another form, namely 1-(1-a)(1-b). Similarly, (a+1/2)2-1/4 is better than a2+a. This is because the so called single usage expressions (SUE, every parameter appears explicitly exactly once) produce tight results, while the more natural forms - overestimate them, i.e. produce wider intervals. The connection with optimization is obvious.
Dear Sergey,
I don't know what means the acronym MLE. Himani Mittal is right saying the objective or cost or goal function is defined by the problem. I believe that your problem is linked with David A. Pelta's statement: the cost function is a measurement to the quality of a potential solution.
I'm guessing that your problem is to find a set of parameters for a numerical model for fluid dynamics of best fitting with experimental data.
If the situation described in the paragraph above is correct, the cost function will be depend what type of method you are choosing to solve the problem: deterministic or statistical one.
A deterministic method to find the best set of parameters: in general, the square difference between the data from the mathematical model and the data from the experiment can be considered a good cost function.
If you want to apply a Bayesian approach (statistical scheme): you are not find a set of parameters, but you are going to identify the probability density function (PDF) associated to the parameters (considered as a "random variables"). In this case, the cost function will be a likelihood function. The most used likelihood function is the Gaussian function (for comparison how good is the agreement between two PDFs).
This is becoming a long comment. Next comment, I will suggest a paper to identify automatically a best likelihood function using an adaptive particle filter.
Dear Haroldo Fraga de Campos Velho, thanks a lot for your answer. Sorry for using of MLE, it means Maximum Likelihood Estimation method.
Dear Marek, once more thanks for your answers. I fully agree with you, that using of Weibull form with (w,b) is prefer. It seems me, that "physically" w (natural scale parameter) is uniform distributed between w_min and w_max, and a ( where a = (1/w^b ) will be distributed essentially not-uniform, most values will be near a_min. It is only one reasons, but I want to formulate several reasons-rules.
To optimize functions, you must first know what the objective to optimize, for example linear functions in linear programming, the life of items in product design using Weibull or lognormal distributions. Thus, the first step is to define the characteristics to be optimized.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Algorithms