I have two estimates, one from logistic regression of different surveys pooled together and the other from a meta-analysis of (pooled estimate from meta-analysis) separate surveys (same surveys used in the logistic regression).
i. estimate (odds ratio) from logisitc analysis= 1.06 ; 95% CI : 1.03-1.10)
ii. estimate (pooled estimate form meta-analysis= 1.02 ; 95% CI : 0.90-1.09)
I want to do a test of significance whether a statistically significant difference between these two estimates exists or not.
The second estimate has a confidence interval that is a subset of the first, thus their difference id not statistically significant.
Dear Dr Alessandro Giuliani
You can use the Students' t-test to check whether it is significant or not.
First, you estimate the standard deviation from 95%CI (available at: https://handbook-5-1.cochrane.org/chapter_7/7_7_3_2_obtaining_standard_deviations_from_standard_errors_and.htm) after that you can write code in STATA below:
1. you check the Variance ratio test:
sdtesti n1 m1 sd1 n2 m2 sd2
n: sample size
m: mean
sd: standard deviation
2. you conduct t-test:
2.1 if Two-sample t test with equal variances (p-value in the (1) >0.05), you use the code below:
ttesti n1 m1 sd1 n2 m2 sd2
2.1 if Two-sample t test with unequal variances (p-value in the (1)
Hello Daniel Bogale Odo. One way to read your question is that the two estimates come from different ways of analyzing (exactly) the same data. Method 1: Compute an effect size for each of k independent groups, and use standard meta-analytic methods to compute a pooled estimate of effect size (i.e., the OR); Method 2: Pool all of the data and estimate a logistic regression model. Is that what you meant?* Thanks for clarifying.
* If if is what you meant, then it is not surprising that the two estimates differ, because the one using pooled data does not adjust for the other variable. Pooling effect sizes (via meta-analysis or regression) and pooling the data are two different things entirely.
You say these are two procedures and are being applied to the same data. So by "significant" do you mean that these procedures applied in this or similar situations will produce identical results. Is this what you mean? If you mean for the population of surveys from which this particularly set of surveys are drawn, you could use bootstrapping. If you mean the later then define the population of surveys. Whatever you mean, it is probably worth defining what you mean by statistically significant difference.
ps. You also don't say how you pooled the data. That could mean using the same procedures as you use in the meta-analysis.
Thank you all! and sorry for the confusion in my question. Please let me clarify it. I have 20 surveys (Demographic and Health Survey) conducted in 20 different countries. I used in Method 1: Multivariable logistic regression by pooling these (20) surveys (merging into one dataset). Method 2: Applying exactly the same model to each survey (country), and then conducting a meta-analysis of 20 separate analyses (studies). Note that I applied the same model, including the complex survey data analysis procedure and covariates adjusted, both for the pooled dataset and the survey (country) specific analysis; as Bruce Weaver rightly stated. Now I want to test whether there is a statistically significant difference between these two (logistic regression & the pooled estimate from meta-analysis) estimates. My main analysis was the logistic regression one and I have conducted the meta-analytic as a supplement and one of my reviewers asked me this: "A test of significance (e.g., Cochrane Q test) would confirm a statistically significant difference between these two estimates". I am not sure whether it is possible to compare these two results or not, and how to go with e.g., Cochran q test in his case. I am sorry for the lengthy text and appreciate any assistance.
This question is nonsensical. Why should you want to show that the data you have are sufficient to conclude that you should expect different results from different methods?
An interesting question was if the to methods are mathematically equivalent. This can be verified or falsified mathematically (logically). And if they are not equivalent, then the results will be different. The difference will depend particular features of the data and the methods and one could even derive what will have the largest impact and if this is or can be of any practical relevance. This is a mathematical work, not a statistical one. You may write the editor that this is beyond the scope of your empirical work but that it might be somethingof interest for a mathematical statistician...
Jochen Wilhelm Thank you very much for this, and I appreciated your suggestion!
You can see from the confidence intervals that there is no significant difference. That being said you can try a t-test paired or unpaired depending on your data.
t.test( ) is the function in R.
- Badges
- Science topic
- Similar topics
- Journal Impact Factor