To fit a specified nonlinear model to data, there are several classical iterative methods available, such as the Gauss-Newton method and the Levenberg-Marquardt method. Several guided random search techniques invented for engineering optimization, such as genetic algorithms, simulated annealing, and particle swarm optimization, also have been applied for parameter estimation in nonlinear models.

A brief review of these method and an approach to find reasonable initial guess for an iterative method could be found in the following paper:

Article Taking the Guess Work Out of the Initial Guess: A Solution I...

Not clear about exact problem. seems interesting...Would a method like Recommender systems using collaborative filter help..just a hunch ....

Good luck

Cheers

If the solution involves integrals not solvable except numerically you can solve the integrals manually or by a software. The problem can now be solved.

Thank you everyone for your interest and input. Just to clarify a little more the problem: the main issue is that the model formula which described the mean response variable as function of the predictor variables and contains unknown parameters to be estimated cannot be evaluated analytically, only by numerical integration. In other words, each evaluation of the function to be optimized (like minimizing sum of squared errors) requires substitution of the parameter values and numerical integration. One "brute force" way of doing the optimization seems to be something like this, a stochastic stepwise algorithm:

1. Choose initial parameter values and store them as best values.

2. Evaluate the optimization function and store it as the best value.

3. Choose new parameter values in the vicinity of the previous best values, e.g. by adding Gaussian random numbers with 0 mean and some selected SD to each parameter value. Maybe do this to only one randomly chosen parameter, not all.

4. Evaluate the optimization function again. If its value is better than the stored best value, update the best parameters to the current ones and update the best function value to the current one.

5. Keep repeating steps 3-4, maybe slowly decreasing the randomness SD.

I think that this procedure should in principle work, but would be slow and may not converge well. I would be grateful for your thoughts about this and/or some better options.

Thanks for the clarity in the problem. It seems a good way to start. Suggest to consider interventions as follows:

1 After step 2 build a matrix ( params output) .

2 use SVD or PCA to identify important linear combinations of

params

3 proceed from your step 3

Alternatively consider ANN

Nice problem ! Good Luck

Cheers

Thank you Narasim for your input. Can you please clarify how would using PCA to identify important linear combinations of parameters help in this case?

Meanwhile I tried the following modification to step 3: instead of centering the Gaussian random numbers at zero, I centered them on the magnitude of the last change of the selected parameter value that led to an improvement in the optimization function. In other words, if during the last iteration where the optimization function was improved the given parameter value changed by +1 unit, then the next random value of this parameter is a Gaussian random number with a mean of +1 and the previously selected SD. This procedure generally works, but does not converge well - some small improvements in the optimization function still occur even after several thousand iterations. Any thoughts on this?

Igor, you need to check KKT conditions in your stochastic approach in order to make sure you get at least a local minimum. I would suggest you to use a classical algorithm (e.g., Levenberg-Marquardt method or trust region reflective method) in a commercial software rather than developing a code by yourself except the data fitting process is the major contribution of your work.

You only have 4-6 parameters in your model. If you have domain knowledge about the range of each parameter or you have abundant computational power (e.g., a server with multiple CPU), you can also try brute force method (exhaustive search in the range of each parameter with specified significant digit).

PCA gives the main linear combination of the matrix(params,output) in its eigen

vectors , with max variation when we choose the two max singular values. We can

detect the param which has max influence with eigen vector components and perhaps change only those for next iteration. This can I think be combined with

max descent approach using your cost/ performance function.Hope it helps.

Cheers

A brief idea of SVD in recommender system is here:

https://www.researchgate.net/post/Does_SVD_reflect_accurately_the_user_ratings_and_movie_classification

I would suggest to use extremely reliable, but brute force, algorithm based on interval analysis. It is applicable when you are able to compute the reliable ranges of response variable for given ranges of fitted parameters - even if only numerically. Outliers (but less than 50% of all data) are irrelevant, nor the assumption of data 'gaussianity', but their standard deviations are needed. When the standard deviations are unknown then all you can get is rather rough, but still guaranteed, solution (or, more precisely, interval hull of solution(s)). The presence of outliers may make narrowing of initial intervals of fitted parameters simply impossible.

More about interval computations at:

https://en.wikipedia.org/wiki/Interval_arithmetic ,

detailed description of the algorithm is given in my paper:

http://arxiv.org/abs/1708.01452

Dear Igor,

Ad 1) we have used smart algoritms based on the so called Genetic Algoritms. We can threat the model as a black box with output data to guess the input parameters.

Ad 2) we have sucessfully implemented very well working robust Bayesian method to omit outliers. Please sent me priv message, I can share it.

Best regards

Kris

With Respect,

There are many methods to solve your problem; however it is important that you can apply this methods. I would suggest a NSGA algorithm to solve your problem.

But it demands a extraordinary knowledge about artificial intelligent.