Dear Naõmi,

I would say both arguments from your supervisor are wrong or ill formulated.

1) The sign of the paths do NOT effect the strength of the indirect effect! The indirect is calculated as the product a*b, so the magnitude is the same, regardless of the sign. If the paths have opposite sign, the indirect effect would simply be negative.

2) Marginally significant a and b paths will surely be a reason that the indirect effect is not significant, as in your case, but this does NOT make the model more complex (this is wrong)! The model is what it is, regardless of the significance. You simply have not enough power in your analyses to reliably detect an indirect effect of .05, neither assessed by the SE (which is .03, multiply by 1.96 = 0.0588, to get an impression of the normal theory CI as rule of thumb), nor by the bootstrap CIs. So, your believe is also wrong, although you cannot directly estimate the significance of the indirect effect from the significances of the simple paths (only as a rule of thumb).

Therefore, you have no special case, but only an underpowered study for your effect sizes. If you think that 0.05 as an indirect effect is of practical relevance, I would discuss that and conduct a replication study with a new sample (do NOT add additional participants unless you are doing Bayesian analyses!!!) with an appropriate sample size in the first place.

The bootstrap test with the confidence interval for the indirect effect is different from tests of the individual a and b coefficients. The bootstrap CI for an indirect effect a*b is typically asymmetric (because the distribution of the product a*b is not symmetric) and tests a different hypothesis than the normal theory t tests for the individual a and b coefficients. Therefore, it is not unusual for the bootstrap CI to yield a "non-significant" result even when the t tests for a and b are "significant."

Good morning, thank you both Christian Geiser and Rainer Duesing for thinking along.

So, as I understand your feedback correctly Rainer Duesing you would suggest that my thesis research is underpowered, and I can argue this by looking at both the SE and the bootstrap CI's? I am not familiar with the normal CI theory, but I will make sure to look it up. Could you support me with a sufficient source that could help me argue that the indirect effect is insignificant due to having not enough power?

Moreover, I ran a G*Power test before collecting data from my convenience sample. This G*Power test showed me that I only needed a maximum of 55 participants within my sample (which is a small number, I know). However, the control variables that I was aiming to add were lacking too much data, so I decided not to use them (that is why I only have 2 predictors). Would you say this G*Power test was correctly done/concluded upon?

See the screenshot of the G*Power test below.

In the end, after filtering out the missing data, I could use data from 83 participants. Thereby, I concluded that this sample size would be sufficient enough when looking at the advice of the G*Power test (minimum of 55 participants).

Thank you again in advance for thinking along.

Naomi Renssen

1) The normal theory approach was just meant as a rule of thumb, to get an impression of the data/significance. As Christian Geiser already pointed out correctly, the distribution of the indirect effect is typically asymmetric and hence, the bootstrapped CI should be preferred.

2) For what exactly do you need a source? If you think that ab = 0.05 is of practical relevance, but your test failed to flag this effect as significant, your study simply did not have enough power. I think this can be found in any textbook. On the other hand, maybe 0.05 is much smaller than your a priori determined smallest effect size of interest (SESOI, you may look up this term). Then you may have a well powered study, but the effect in the sample (and therefore the inference on your population) is smaller than that and would have been correctly flaged as non significant. But this is up to you, to decide that.

3) I think your G*Power approach is wrong on several levels. a) G*Power assess the increase in R^2 for one regression model, but what you want is a indirect effect, which cannot be assessed via G*Power. b) I don't know how you came up with a f^2 of .30, but how does this correspond to the magnitude of the indirect effect ab = 0.05? I mean, I would find it difficult to translate Cohens f^2, an estimate of the explained variance of the model, into a (product of) regression coefficients, which are in the metric of your dependent variable. Maybe you did that (and I would be glad if you could share that), but clearly, .30 is a rather large effect size (for a simple regression model) and I would expect indirect effects to be smaller per se (since they are a product of imperfect predictions). c) Additionally, a mediation model is more complex than a simple regression, therefore power analyses for simple regressions may be misleading. A simulation approach, which incorporates a mediation model may be more suitable and there are online tools for it. d) control variables may be good or may be bad, you should use directed acyclic graphs (DAG) to check which variables to include or not.

4) What do you mean by "filtering out the missing data, I could use data from 83"? How does the sample size increase, if you "filter out" missing data???? And why didn't you use that sample in the first place?

Thank you again Rainer Duesing ; this input helps me a lot to critically reflect upon the choices I have made.

2) Since researchers in social science usually use the confidence interval of 95 percent, I was recommended to use the cut-off score of p < 0.05 as well. Thereby, I would argue that using .05 is of practical relevance. When looking at my regression results (see the first screenshot of the regression table) I would conclude that ab is non-significant, since zero has been found between the BootLLCI and BootULCI (CI [−.00, .12]). Thus, I think I will simply discuss that my power is not enough, based on ''a textbook''.

In addition, I am indeed not familiar with the alternative argumentation, in relation to SESOI. Nevertheless, based upon my argumentation above I do think .05 is a relevant cut-off score (and not too small) since this is usually the standard in social science.

3) Hm, honestly I have just followed the instructions that I got from my supervisor and thus used the G*Power tool and these settings (incl. f^2 .30).

b.) It is interesting to read that you would say that an f^2 of .30 is a rather large effect size. Based upon what rule(s) of thumb would you argue that this is a rather large effect size for a simple regression model, including an indirect effect?

4) I think we have a bit of a miscommunication about the sample size. To make it clearer; I conducted my data with five other thesis students. So our survey included all kinds of different scales (thus also irrelevant scales/data for my research). In total, 190 participants have filled out our survey. However, after deleting the missing cases based on a listwise procedure only 83 cases were left in my dataset to use within my analysis. The G*Power tool gave me the advice to at least collect data from 55 participants, and therefore I concluded that using data from 83 participants would generate enough power for my study to draw reliable conclusions (since 83 participants meet the G*Power criteria of having bigger sample size than 55 participants).

Naomi Renssen I think you did not get my point and argument about the .05! It is not about the significance threshold, but your effect a*b = .05, as can be seen in your first output. There is no need to talk about your significance level, but the size of your indirect effect, which is 0.05 in the metric of your depndent variable. So your argumentation about your "cut-off score" seems to be wrong. Think about the practical relevance of your indirect effect in terms of changes in your dependent variabel. How much should the indirect effect change the dependent variable, if your change X/independent variable by 1 unit. That's the issue here, not 95% confidence.

If you have honestly followed the instructions, but did not understand what you are doing and what the .30 means, nor if .30 is considered small, medium or large, I simply cannot help. You should know what you are doing.

All effect sizes are up to discussion, what may be small in one research field may be large in others. Whithout knowing anything about the content of your study, I could only guess. 1. G*Power itself gives rules of thumb (following Cohen), and .35 is deemed large. 2. An f^2 of .30 would mean that with a residual variance of 1, the variance explained by the special effect is 30%, which seems large. I have no idea, how much an indirect effect should explain. As I said earlier, I find it unintuitive and a better approach might be to simulate the power with the relevant parameters that you expect or even better, that are the lower bound of practical relevance --> SESOI. The concept of "explained variance" is nice, but is this really what you want to know? Do you really care about explained variances or how much your independent variables change your dependent variable, which would be ansewered by regression coefficients (and hence the indirect effect, which is simply the product of regression coefficients). Ask yourself: how does the f^2 of .30 that you took for the power analysis, connect to the a*b = .05 in your analysis? Is .30 larger, equally or smaller than .05? They are completely different metrics, so please explain to me, how you compared them?

So you have an actual sample size of 83, which is larger than your power estimation, but you still did not find a significant effect, then you may conclude either two things 1) your effect of interest is much smaller than your expected effect size (according to your power analysis) or 2) or your effect size was inappropriate in the first place (what I would assume, too be honest, unless you can explain the connection/translation from f^2 = .30 to a*b = .05)

Rainer Duesing I indeed misinterpreted your previous comments, and I understand your point now. Moreover, I knew that 0.30 was considered a large effect size by Cohen, and I do think it is important in my study to be able to conclude upon the explained variance. This mediation model has never been studied before and therefore I think it would be interesting to be able to explain how much of the variance of the relation between X and Y was caused by the indirect effect (mediator).

In addition, the slope of a*b is relatively small (0.05) however, I do believe this is of practical relevance to the existing literature. This mediator explains a part of the relationship between X and Y. Thereby, I could argue that there are also other mediators that explain this relationship (based upon prior research). Would this be an appropriate argumentation when explaining why I think this indirect effect is (small but still) of practical relevance?

Furthermore, I do not know how I would be able to sufficiently explain the connection between f^2 .30 and a*b = .05. Would you be open to explain the relation between these two metrics, and how these metrics could be linked?

Lastly, I could discuss in my discussion & conclusion that I think that 0.05 as an indirect effect would be of practical relevance, however, due to an underpowered study it was not possible to conclude upon a significant effect. Would you say it would be appropriate to advise future research to conduct a replication study, with a new sample and a higher power (lower effect size, and higher sample size)? And connect this advice to the fact that the effect size (0.30) might have been a little too large/overestimated.