For example, there are many self-report instruments for assessing depression. Can effect sizes generated from studies employing different measures of depression be pooled to provide an average effect, or should pooling be restricted to effect sizes generated by studies employing the exact same instrument?
In most programs, such as Comprehensive Meta Analysis, it is relatively straight forward to conduct moderation analysis. Consequently, you can include every available measure of depression and conduct separate analyses for meaningful groups of measures. The program will provide you with an overall effect size, effect sizes for each group and several indicators of heterogenity which show whether or not det established effect sizes are different. While it deals with prevalence rates of bullying rather than depression, you may find the following paper useful with regard to understanding how these moderation analyses work. https://www.researchgate.net/publication/229696433_The_impact_of_methodological_moderators_on_prevalence_rates_of_workplace_bullying_A_meta-analysis
Article The impact of methodological moderators on prevalence rates ...
Thanks everyone for their answers, they've been very encouraging. My question was a little disingenuous. I have used meta-analysis on a number of occasions (it formed a central part of my 1995 PhD thesis, which very much argued for its use as an exploratory tool), but not very much in recent years. One thing that I have noticed that in the reviews conducted by NICE and Cochrane the effect sizes appear to disaggregated down to the level of the study or variable/instrument and this appears to be the position prior to any heterogeneity analyses being conducted.
For example, NICE has put out for consultation some guidelines on the management of low back pain, with or without sciatica (see here https://www.nice.org.uk/guidance/GID-CGWAVE0681/documents/draft-guideline). I am particularly interested in the evidence for psychological therapies. However, inspection of their evidence tables show, for example, that effects are separated into separate analyses depending on the type of measure involved. On page 583 (it's a monster of a document), for example, the evidence for the effects of CBT in comparison to behavioural therapy, on patient function, is summarised. However, the evidence is reported separately for different measures of patient function - specifically, the Roland Morris Disability Questionnaire (RMDQ) and the Quebec pain disability questionnaire. This is repeated throughout the document (the next page splits results from two different measures of pain intensity). As well as disaggregating effects by different measures, there also seems to disaggregation of effects based on some apriori notions about what comparison groups should go together and which should be analysed separately. For example, on pages 168-9 the effects of self-management compared to exercise is summarised in one table and the very next table the effects of self-management to yoga is compared. I thought yoga was exercise! It seems that throughout this document there are a huge number of decisions made in an apriori fashion, leaving only one or two studies in each of the multitudinous categories from which it is able to draw conclusions. Unsurprisingly, this results in weak evidence and low confidence. My impression is that it is not just this review but a general trend that seems to go against the ethos of meta-analysis as a rich exploratory tool.
I did think for a while that I was behind current reasoning in meta-analysis, but your answers have reassured me. Or perhaps I'm missing something? Grateful for further comment and opinion.
Dear Martin,
i recently experienced the same. I followed the methodology applied by P. Cuijpers in his studies, because he is a recognized expert in meta-analyses, and combined the estimates of different outcome measures assessing the same concept. I received lots of criticism on doing that as i was explained that recent trends in doing meta-analysis are oriented towards reporting pooled effect sizes from only one instrument or comparing the results from two or more instruments. Of course, this was the opinion of two reviewers, not a general point. But for me that doesn't make sense as meta-analyses in general have an exploratory nature. But obviously things are changing. Anyway, i did a sensitivity analysis assessing the impact of each instrument on the overal effect size and this was accepted. I do think that after all everything depends on your way of justifying your methodology, but you are right that trends in meta-analysis are changing and many people think that mixing "apples and oranges" doesn't provide robust evidence anymore. I will continue doing meta-analysis in the old fashion way, though, because for me this provides more interesting information than restricted pooling of effect sizes generated by studies employing the exact same instrument.
Thanks for the helpful replies Kaloyan and Maryna. I still feel that we are losing something (statistical power, for one thing) by analysing effects from different outcome measures separately on an apriori basis, when it wouldn't be too difficult to examine whether these different measures actually make a difference to the size of effects. Off to do more reading.
Rohit, wouldn't you run into another problem then? That is, that your study is not powered to find measurement instrument-effect. This might be a problem when the research field is fragmented with lots of instruments in use (common in some psychology domains), or when doing study on an area with only a limited number of studies.
Thanks everyone, that's brought me back up to speed a little. I'm firmly in the 'pooling' camp, but it looks like I'm swimming against the tide. Perhaps it will change back again in the future.
- The causes are bio-physical, emotional, affective, social and environmental and circumstantial.
- The most appropriate assessment would be with rating scales of self-esteem and self-concept.
- The criteria for such assessments, however, remain relatively subjective, since they depend strongly, on the human, scientific and technical formation of the psychotherapist.
- The effects of the treatment will depend, however, largely, on the commitment, maturity, responsibility and technical and scientific involvement of the psychotherapist.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis