I am carrying out a meta-analysis to investigate the relative degree of impairment for subcomponents of executive functioning for people after brain injury (ie. cognitive flexibility, inhibitory control, working memory and planning/organisation ability).
Some studies report mean and SD for two or more measures of a given aspect of Executive Functioning (eg. two measures of inhibition). Given that they are conceptually similar, can you somehow combine these two means to get a pooled effect size for that study?
Following this, can you and how, do you combine this effect size with the effect sizes from other studies included in the overall meta-analysis.
Any advice would be greatly appreciated,
Thank you!
Mary-Kate.
Of course you CAN. The questions are more along lines of: SHOULD you? And if not, then how will you WEIGHT them in the final meta-analysis?
In favor of combining: you're getting what may be a more reliable measure.
Against combining: they may not really be measuring the same construct. Just because they both purport to measure "executive functioning" doesn't mean they have that much in common. I'd suggest checking on studies that include zero-order correlations between the tasks in question. If they consistently correlate at (say) .50 or above, then sure, they're getting at the same thing, but if they tend to correlate at (say) .30 or below, then maybe not.
If you don't combine them, you want to take steps to prevent that single study from carrying too much weight. If there are (say) 80 people in the sample, then reporting 3 different effect sizes from the same sample would give it the same weight as a study with 240 participants, which you probably don't want to do.
Whatever you decide, of course, be sure to report clearly what you did. That way your readers know how the meta-analysis was constructed.
You can include multiple effect sizes per study/sample if you carry out a 3 level meta-analysis (or commonly known as multilevel meta-analysis). Its one way of 'combining two or more measures' within a study, without you having to make any arbitrary decisions on the weight of each within study effect size. This 3 level meta-analysis does not require the assumption of independent effect sizes since an intermediate level is included to model the dependence between effect sizes within a study.
You can easily run a 3-level meta-analysis in R using package 'metafor' . There is a really helpful tutorial here https://pure.uva.nl/ws/files/26092656/p154.pdf if you want to do that
Thank you Stephen and Junhong for your helpful and informative answers. I am so grateful. I will read up some more on both suggestions.
Mary-Kate.