For example, the same model with 2 parameters is fitted to two data sets. Confidence intervals for each parameter are estimated by nonparametric bootstrapping. The results define two separate (one for each data set) 2-dimensional confidence regions for model parameters (each bootstrap gives a combination of 2 parameter values). The region overlap to some degree. What is the best way to estimate the p value for this overlap? In other words, at what level of significance do the parameter confidence regions generated based on each data set differ?
Why don't you directly bootstrap the difference in the parameter values?!
Thank you, Jochen! maybe I misunderstand your point or did not describe the task clearly. Perhaps the attached data set will help. Here the same model with 2 parameters was fitted to 2 different data sets, and parameter confidence regions estimated by bootstrapping. When the two regions are plotted (x axis is parameter 1 and y axis is parameter 2, see attached file), they overlap somewhat in 2-dimensional space, but less so than the ranges for each parameter - because the two parameters are correlated. I am wondering how to estimate a p value for the overlap. Would be grateful for comments from you and anybody else who may be interested!
I wonder, does the following procedure make sense for the problem described in this question:
1. Suppose fitting the 2-parameter model to each of 2 data sets produces best-fit parameter combinations x1,y1 and x2,y2, and nonparametric bootstrapping generates multiple parameter combinations x1(i),y1(i) and x2(i),y2(i)
2. For each bootstrapped parameter combination calculate the distances d1(i)=sqrt((x1(i)-x1)^2+y1(i)-y1)^2) and d2(i)=sqrt((x2(i)-x2)^2+y2(i)-y2)^2)
3. Remove a certain percentage (e.g. 5%) of bootstrapped parameter combinations with the largest distances from both distributions: d1(i) and d2(i).
4. What remains are 2-dimensional 95% confidence regions for model fits to both data sets, and they can be examined graphically whether they overlap or not.
I would be grateful for feedback from anybody interested!
Hmmm, as to my gut feeling, the "distances"-approach is not ideal. It may work for particuar correlation structures, but I am not sure about its general interpretability.
I would go for the analysis of all the data (both datasets) together and use "dataset" (factor with two levels) as additional predictor. Then you may analyse the interaction of "dataset" and your two parameters. This directly answers your question, how strong the values of the parameters depend on the dataset. From the plot in your file I would assume that parameter1 shows some interaction (higher for dataset2), whereas the value of parameter2 does not depend on the dataset.
This is interesting. Please keep me up-to-date.
Thanks, Jochen! What you are saying is reasonable, I will try it and post the resulting data file.
Hi Jochen (and anybody else interested),
Please see sheet2 of the attached Excel file. There bootstrapping was done on both data sets with interaction terms added for parameters 1 and 2. The results suggest that indeed parameter 1 is higher for data set 2 (p value ~ 0.055), but parameter 2 is the same for both data sets (p value ~ 0.58). Does this kind of analysis make sense?
I still have a mental problem with the correlation of the two parameters. Your bootstrap data also shows an obvious correlation of the parameter and the interaction. This already needs to be understood. I have no good idea what this means. Further, given this correlation, then if the two parameters are correlated as well (and this is the case as you said), then the interaction is correlated, too. Therefore, a conclusion like "parameter 1 depends on the data set, but not parameter 2" is nonsensical. There is only a difference in the magnitude of the dependency.
Thanks again, Jochen! Interesting comments about the correlations. Are the interaction terms really meaningful then or not? In other words, do they help answer the original question about parameter differences for fits to different data sets?
Exactly this is the point, and I don't know the answer. It may be irrelevant or simple, it may be important and complicated, too - I just dont have the brains to judge it, sorry.
Perhaps then a conceptually simple solution for this 2-dimensional problem would be visual inspection of the region of overlap between the "clouds" of bootstrapped parameter combinations for fits to both data sets (shown in sheet 1 of the attached file)? For example, if there were 10000 bootstraps for each data set and say 50 points fall into the region of overlap, the p value for differences between fits to the two data sets would be 50/20000?
Well... what I don't understand is the null hypothesis. You have different datasets, and you simulate the null distribution of the parameters for these datasets. These are actually two null hypotheses, and I just have no idea if the overlap of two null distributions is a "p-value".
Maybe I am not interpreting your response correctly, but perhaps this may help to clarify the situation: the independent variables in both data sets are identical, only the dependent variables differ. So the null hypothesis would be then that the response of the dependent variable (modeled by the 2-parameter function) is the same for both data sets, i.e. the "clouds" of bootstrapped parameter values should overlap. Is this reasonable?
Thanks again, Jochen! So does it make sense then to assume that the fraction of bootstrapped parameter combinations from fits to both data sets which fall into the "region of overlap" estimate the p value relevant for this null hypothesis?
- Badges
- Science topic