What you are describing is called "effect size dependency" and it occurs when multiple effect sizes (for you, correlations) are reported from a single study. More specifically, dependency occurs when there are multiple outcome measures using the same groups, or multiple outcomes from several measurement points (as in a longitudinal study). There are other sources of dependency, but those are the common ones. And you are correct that you should not just include two dependent correlations from one study in the model like you would any other correlation. If you did that (i.e., ignored the dependency), you would artificially reduce the standard errors of your model output, and therefore potentially misrepresent significance. This would occur because your model would treat those two correlations as independent, and therefore they would seem to provide more information than they actually are. You're also right about the data loss that would come with excluding the studies reporting multiple effect sizes - you want to avoid that if you can.

You have a couple of options, and I'm going to assume that you are working with random-effects meta-analysis for this discussion. The first and most straightforward is called robust variance estimation (RVE). In this approach, when you're setting up the model you also specify the study ID that effect sizes are "nested" within; the model then takes into account the nesting in estimating the standard errors. With RVE, those studies that report multiple correlations will be correctly incorporated into the model, so you'll get the best of both worlds: all the information from your studies, and correctly estimated standard errors. You'll also get fairly standard meta-analysis output that you can go on and interpret as usual. Your other options are multivariate meta-analysis and three-level meta-analysis. I won't get into the details, but I think these methods are probably unnecessarily complex for your situation, so I would argue that you should just go with RVE.

You didn't mention what platform you are using for analysis, but RVE is available in STATA (Article Robust variance estimation with dependent effect sizes: Prac...

Study_ID......Cor_type......Cor

1...................Indiv.............0.7

1...................Group...........0.5

2...................Indiv.............0.9

3...................Indiv.............0.2

4...................Indiv.............0.4

4...................Group...........0.3

This is incredibly helpful! Thank you so much! Nicholas J. Parr

So it is going to be Indiv, Group, Mixed as Cor_type. But this indeed answers my question perfectly.

Seems like I will not be able to avoid learning R for this. ;)

That's correct - and you should learn R anyway!

A couple of other thoughts. No matter what method you use, the dependency is addressed by including a correlation between the dependent effect sizes in the model (in your case, a correlation between correlations). Most primary studies don't report correlations between their effect sizes, so part of the challenge in dealing with dependency is how to estimate or decide what that correlation should be. I say this because robumeta (the R package for RVE) by default assumes 0.8 as the between-effect size correlation, which is something you should report in your article. Because this is an assumption, there is a function called sensitivity() in that package that runs a sensitivity analysis on your model by testing a range of values for that correlation; you should run that and also report whether you results differed substantially for different values of the correlation.

Second, the Cor_type variable is mainly for you to keep track of what type of correlation was reported, but after you do all of your main analyses, I would do two things with this variable to help you understand your data/findings better. It seems to me that when studies are reporting individual or group, that's substantive information about how the intervention was delivered, but "mixed" is instead saying something about how the outcomes were reported. So, I would define two new variables using Cor_type, and include those in meta-regression models.

For the first one (I'll call Intervention_type, I would do three levels: Individual, Group, and NA (missing) for Mixed. Then, if you include that in a meta-regression model as a covariate/moderator, it will tell you whether interventions implemented in individual had a different correlation with the outcome than interventions implemented in a group session. You will lose the correlations reported as Mixed from this model, but this is a substantive question that seems valuable to explore regardless.

The second one is more of methodological or sensitivity test related to reporting. I would create a variable with two levels (called Report_type): "Unpooled" for the correlations from the Individual or Group levels of Cor_type, and "Pooled" from the correlations reported as Mixed/pooled at the primary study level. Including this variable in a meta-regression will tell you if there are systematic differences between correlations reported from single interventions or from correlations pooled across intervention formats. If you did see moderation, it might tell you something about potential reporting biases.

So, your model formula for the three model will be:

Cor ~ 1 (your main model, which is just the intercept, or the mean correlation across your studies)

Cor ~ Intervention_type (meta-regression testing the moderation effect of Individual or Group intervention format)

Cor ~ Report_type (meta-regression testing the moderation effect of whether the effect size was reported in its original form or collapsed)

I am blown away. I did expect some good answers here on researchgate but your help exceeded my expectations by far. Thank you very very much.

Nicholas J. Parr - I do have a follow up question and I hope that I am not overdoing it by tagging you directly. I hope you'll find the time and patience to answer them. For several reasons I am questioning the usefulness of a metaregression with my dataset. The reasons are listed below and I just wish to check if my reasoning is completely mislead here.

1- When the sensitivity analysis reveals similar rhos it can be said that the pre-specified rho (it is 0.8 and not 0.7, btw) is good to use. Is that correct?

2 - The results of my main model suggest that there is no sizeable and/or significant effect to be found.

Given the small number of total studies and total endpoints / effect sizes in my analysis I wonder if it makes sense to perform a meta regression after all? This is a thesis and not supposed to be published, hence I assume that a result saying the treatment doesn’t seem to be related to the outcome is also a sufficient result to answer my question.

Results of my main model -

Model: effect.size ~ 1

Number of studies = 12

Number of outcomes = 16 (min = 1 , mean = 1.33 , median = 1 , max = 3 )

Rho = 0.8

I.sq = 73.19794

Tau.sq = 0.01984602

Estimate= 0.0528

StdErr = 0.0487

t-value = 1.08

dfs = 10.4

P(|t|>) 0.303

CL L -0.0551 CL R 0.161

3- Nevertheless, I tried to implement your suggestions just to see if I can work the package in R in general. First I noticed that robumeta cannot handle the NA values. It returns an error report. This is probably an issue with me being new to R, so I might need to just learn how to work around that. HOWEVER - I ran into a couple of issues, which make me think that I cannot perform a meta-regression. I might also want to report to explain why I did not perform a meta-regression. I’d like to know if my reasoning is correct here.

a) Re: Interventon_type

Coding: Group=1, Individual =0, Missing or Mix = NA

• Theoretically: There are studies which did not report delivery type at all (already coded as NA) and also did not provide the delivery type after sending an email for clarification. Reporting of delivery type by endpoints (k_16): Group k = 5 Individual k = 3 Mix k = 5 Not Reported k = 3
• This results in df < 4 in all of the separate cases. Robumeta prints: „Note: If df < 4, do not trust the results„.
• Hence I cannot perform this kind of analysis, if I understand correctly.

b) Re: Report_type

• See above, I end up with three variables (pooled= 1 undpooled =0 , Nor Reported =NA)
• This means that, again, the df for pooled is too small and the results cannot be trusted

1 - That's correct.

2 - Correct on the interpretation that the summary/overall correlation is minimal and non-significant.

3 - Based on the added information you provided about reporting issues and the number of studies you have, I think your conclusion to not use meta-regression makes sense. It is not uncommon to encounter degrees of freedom issues with RVE, especially with meta-regression and a small number of studies, so that is an adequate justification.

You will want to speak about heterogeneity when you present your results. It may be worthing discussing the I.sq (I2) value. This is a "relative" heterogeneity indicator (see this recent paper of ours Article Meta-analytic approaches for examining complexity and hetero...

Dear Saskia,

Dear Nicholas,

Since sometimes I am reading and following the discussions about the M A - issues you both are talking about. I do this, 'cause I am facing the same situation so often. The advice given is truly valuable and I am using the prediction - Intervals - xls dialogue sheet now aswell the CMA software.

I am very glad receiving this unigue opportunity to see clear through my blurry minds, while questioning my results of my MAs. Makes the nights of over time shorter and more efficient.

For this thank you again!

Regards

Jimmy Verstege

This link maybe useful for you :

https://www.sciencedirect.com/topics/neuroscience/meta-analysis