I would like to compare some studies using the Hedges'd (or its unbiased estimate g). but in both case i have a hierachihcal structure of the data related to transect representing a secondary level o sampling which correlate some of the observation. I was wondering to simply apply an ANCOVA using this transect as a covariate (could it be done?). It would be surely better to use a mixed model with transect as random effect factor, but in this case may I us this Hedge statistics? How could i adjust for the random effect factor?
I use R, thus lme4 package for mixed models and the compute.es for the effect size
Dear Lorenzo,
It is just an opinion, but I would continue to use d/g even when the result comes from a mixed model. The point of effect size statistics is to be able to compare and contrast results from different models. The point of most mixed modelling is to correctly partition variance so that various types of effects can be accurately determined. If your mixed model shows that some higher level variables are important, then you are "proving" that variability in your dependent measure is better assessed when they are incorporated. On that point you can consider your mixed model to be representative of a much larger unobserved space in which the higher order variables average out and do not bias the effect of interest.
I would not "adjust" for the random effects beyond simply reporting the effect as estimated by the model you believe to be most justified.
Dear Bruce,
Really thanks for your opinion (sharing opinion is always good). I also think that it would be better to use the effect from the more robust specification of the model. Just to talk about it, I realized that I have several options. To calculate the Hedges' statistics I could use the F value from the model, or (according to this document http://www.psy.mq.edu.au/psystat/documents/standardised_effect_size_in_mixed_ML_models.pdf) the marginal means and their SE. I made some try and I am obtaining quite different results. For instance surprisingly I am getting a not significant effect size using marginal means and a significant one using the F value (for the same data!). I would like to know your opinion on that.
Also I am not completely sure about the reliability of marginal means (or least square means) and their SEs, as I know that there is some disagreement regarding their use.
cheers
Lorenzo
Dear Lorenzo,
I am not sure why you would obtain "quite different" results. They should be very close to the same. In totally balanced designs and typical linear model I expect them to be the same. If you have covariates and unbalanced n's then they should be "near" the model specification. Both of these assume that the model specification is sound - so maybe this is telling you about a problem.
How far different is the SD from the two methods? Do you have the sort of joint distributions and homoscedastic residuals expected for all the variables involved?
I might start to debug by redoing with the simplest model and its marginal means, then put in 1 level 2 factor and redo, etc..
Good luck
Dear Bruce,
Again thank for your help... just to discuss further my case...
No problems with normality and homoschedasticity (i have a look to the residuals and they look good)
i think that my problems may be related to what the marginal means actually are and to their reliability... i made some tests: I started from a simple anova (without any specification of the random factor). In this case (as I was expecting) the effect size calculated with the groups means and their SD and those obtained from the anova F value are exactly the same (same value and same SD). Wthen i tried to fit the mixed model.This time using the F value from the mixed model the effect size was still highly consistent with the previous results (values anbd SD were just slighty differnt). On the other hand i had totally different results using the marginal means and their standard errors. This tests directed me to avoid the use of these latter in favor of the calculation with F value in case of mixed models (it seem to me that it should still be robust).
Dear Lorenzo,
It makes perfect sense to me that the model F would be reasonable. What makes me wonder is the case where it does not work - normally this sort of "surprise" tells me that I am not estimating what I think I am. If I think about marginal means being inappropriate it might mean that the joint distribution of my variables is such that I am introducing a problem if I ask the software to "pretend" that the mean estimate can be done over the average values of everything else. One example of this is an interaction between the variables that is not in your model; another could be partially connected error variance.
Anyway, how far you wish to go with this given that you already have a good answer is up to you. I will think about it!
Dear Lorenzo,
I wonder if you finally applied the mixed model with the confidence intervals and how did you compute it, since I am not sure which is the best approximation to do that.
Thanks in advance,
Julio.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression