I would be grateful for your opinions about the following methodology:
Suppose there is a data set consisting of multiple data points with error bars, e.g. in the format X, Y, standard error(Y). Suppose there is a function of f(X) (e.g. linear or non-linear) with parameters a, b, c, ... which needs to be fitted to this data set, and uncertainties for each parameter need to be estimated. The proposed method is:
1. Generate multiple Monte Carlo simulated data sets from the original data set, using the point estimates and error bars and assuming that the errors are Gaussian (or, if needed, conform to another distribution).
2. Fit the function f(X) to each simulated data set.
3. Estimate uncertainties (e.g. 95% CIs) for each parameter of f(X) from the distributions of best-fit parameter values.
The technique you describe is a Monte Carlo simulation of synthetic data. It is a powerful technique that is quite costly but that will work fine provided you know the correct distribution of the errors.
You might want to take a look also at the so-called Monte Carlo bootstrap method as well as the various techniques for estimating the confidence limits in the parameter space (see for example, Numerical Recipe §15.6).
The technique you describe is a Monte Carlo simulation of synthetic data. It is a powerful technique that is quite costly but that will work fine provided you know the correct distribution of the errors.
You might want to take a look also at the so-called Monte Carlo bootstrap method as well as the various techniques for estimating the confidence limits in the parameter space (see for example, Numerical Recipe §15.6).
You may do this taking into account such requirements:
1. You must obtain a very large number of data sets.
2. Your data set must contain independent values with given law of distribution.
It will be very important to characterize the error distribution - particularly any non-Gausssian tails as they will drive the model fits and hence your confidence limits. This Monte Carlo approach will probably produce larger confidence limits than the techniques based on formal error in the fit parameters referenced above by H.E. Lehtihet.
Thank you, everyone, for helpful and useful answers!
Vladimir, could you please clarify your point 2? The data set presumably would be experimentally derived, so its true distribution will be unknown - only assumed.
Mark, could you please suggest some techniques for characterizing the error distribution? For example, the experimentally-derived distribution may pass normality testing, but perhaps it can be described more accurately by a different distribution (e.g. with longer tails)? Also, what kind of data sets are most/least favorable for this kind of analysis?
Thanks in advance.
What is described here is a conditioned parametric bootstrapping approach. The common way to do this is:
(1) obtain the expected (fitted) values for each observation and most likelihood estimates (MLEs) for the parameter by fitting the function f(x) (by minimizing the sum-of-squares or log-likelihood)
(2) if a Gaussian error model is assumed, it is common practice to generate random residuals from e ~ N(0,sigma^2) (i.e. as a function of a mean = 0 and sigma), where sigma is the residual standard error of the model given by sigma = sqrt(Sum-of-squares/n).
(3) the vector of random residuals is added to expected values to obtain a vector 'new' random observations
(4) The function is fitted to the random Observations given the variables in the dataset and the estimated parameter values are noted
(5) repeat this at least a 1000 times
(6) 95% Confidence intervals are typical obtained by taking the 2.5th and 97.5th percentiles, respectively.
This approach is often used to obtain informative CIs for non-linear models.
A commonly used alternative error model is to assume that the residuals are log-normally distributed. This can be easily evaluated by standard model validation graphs (Frequency distribution of Residuals, Quantile-Quatile plots etc.).
In general, parametric bootstrapping is a very powerful and flexible technique. In pricipal, random vectors of 'new' observation' can be generated from any given distribution (as a function of the expected values and the variance function), including Poisson or negative Binomial for count data, Binomial for binary (0/1) data, gamma for right-skewed data and even the Tweedie for compound mixture error models. Statistic platforms like R have the corresponding number generators readily available, but even the poptools add-in Excel provides a selection of different number generators.
I suggest having a look at the following sources:
Efron, B., Tibshirani, R.J., 1993. An Introduction to the Bootstrap. Chapman and Hall/CRC, Boca Raton, USA.
Efron, B., Tibshirani, R., 1986. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science 1, 54–77
Efron, B., 1985. Bootstrap confidence intervals for a class of parametric problems. Biometrika 72, 45-58.
Thanks a lot, Henning, for useful suggestions! But can you please clarify what is the advantage of generating random residuals using best-fit model predictions, instead of generating random data sets using say the observed means and standard errors?
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis