Hello Ross,

There are two reasons why this apparently paradoxical (it's not, really) result could occur.

1. The test of R2-change = 0 for the second stage of your hierarchical model has both a different hypothesis (an equivalent hypothesis statement is that b5 = b6 = b7 = b8 = 0) and different sampling distribution than the test for whether a single regression coefficient (b6, for the A x C term) is different from zero.

2. The sample size was "marginal," given the magnitude of the (largely A x C) effect, thus there was insufficient power for detecting differences (change in R-squared) of the magnitude you observed.

Does this happen? Yes, as your data set would clearly indicate. How to prevent it? A priori, set a defensible target threshold for R2-change as your target ES, and evaluate the requisite N (given that you're testing a second-stage model which adds 4 IVs).

How to explain? See above, and don't forget to warn future researchers about what sort of magnitude of interactions (evidence for moderation) they might expect, based on your study findings. You may, of course, judge whether the observed degree of change is noteworthy or negligible, from a practical / clinical point of view.

Good luck with your work.

I agree with David Morse answer and would like to add the following: you have to keep in mind that you not only add explained variance to your model with significant or nearly significant predictors (no discussion about the use of "significance" intended) but also model complexity, which is accounted for by the degrees of freedom. You added lots of variables, which add more complexity than information, which results in an non-significant overall model. Maybe it is worth to also look at information criteria.

Additionally, it may be helpful to see the output of your analyses to get a better understanding.

The most likely issue is that the A x C interaction is only just significant and the other interactions just add noise. This means that the model 2 R^2 change is overall not adding much information. The interaction effects may also to some extent be collinear - making it harder to detect any effects. You could have a suppression effect if the A x C interaction is not present when other interactions are in the model, but that's not too common.

Rather than dummy code A it may help interpretation to effect code (using -1,1 or -0.5, 0.5 values) as this can aid interpretation of moderator effects. The overall models aren't impacted. It is also important to centre B, C, D and E before computing the product terms for the interactions if doing this manually).

Thank you all so much for your feedback.

David Morse, I hadn't considered the fact that I was testing different hypotheses with each new model. But with some reading on the subject, I think I now understand what you mean. I was curious though about setting a defensible threshold for R2. I don't think I've seen that before. Is there a paper or book that comes to mind for me to read up on this?

Thom Baguley, I'll try effect coding to see what happens. I was going by Aiken and West (1991), but I may have misunderstood the text. Fortunately, all continuous measures were centred, again because of Aiken and West. Initially I hadn't centred them and the results were even more confusing.

Rainer Duesing, your comment about adding complexity is what I was affraid of, though I didn't know how to articulate it. I have several moderators that I need to test. Is HMR not recommended for this?

I've changed the analysis slightly.

Step 1: Condition (Dummy Coded)

Step 2: X2-X9 (Centred Moderators, but not as interaction variables)

Step 3: X1xX2... X1xX9 interaction variables.

I'm guessing this is going to add too much complexity? I'm half tempted to use Hayes' Process for each individual moderator. But with all of the additional tests, my Type I error would be through the roof, and a Bonferroni would reduce my alpha considerably. Oh, and I should add, N = 164.

Ross Anon model complexity increases with the amount of variables in the model. In the regression model this is penalized in the F-test (MSSQmodel/MSSQresiduals, where the MSSQ are the the Sum of SQuares divided by their degrees of freedom(df), respectively) by increased df for the model (so MSSQmodel gets smaller) and by decreased df for MSSQresiduals (gets larger), so the the F-ratio gets smaller, relatively speaking. If the additional explained variance (repredented by SSQmodel) does not outweigh this penalization, you will get non-significant results (a little bit oversimplified to get the point, I hope).

In your case, you entered lots of variables and the increase in explained variance is quite small. Don't do it, it's p-hacking, but if you for example would have only entered X4 and X1xX4, you might have gotten a significant increase in R^2.

Another point is that I am pretty sure that all of your predictors are correlated with each other. The tests for R^2 and change in R^2 account for all explained variance, but the test for each single predictor does not, only the unique variance is tested (semi-partial correlation). So, it can be the case that the overall model is signficant, but (nearly) all predictors are not significant.

You didn't say anything about it, but do you have hypotheses for each variable or are you looking exploratively for moderators? Did you do an a priori power analysis? I do not have a solution for your problem, but p-values are not everything. How would you interpret the regression coefficients?

Hi Rainer Duesing

Thanks again for the feedback. I think you've nailed it, collinearity was the issue. Looking at the VIF scores some were over 7 in the original model, and from what I've read this past weekend 7 is concerning. Having now read up on the issue, the general consenus is to drop the variable that's causing the problem (X3). Unfortunately, and to answer your question, the study's exploratory so I need to test each as a moderator.

Fortunately, there's theoretical grounds to reduce complexity, and so I've taken your advice and done just that. X3-X6 form a larger composite measure (not included in the regression), and so I tested those separately from the other Xs. However, I think collinearity is maybe still an issue. At Step One, X1 (dummy coded condition) and X3-X6 were entered. At Step Two, the interaction variables. Again, Model 2 Sig. F Change wasn't significant, but contained a significant interaction for X1X4, while Model 1 contained a sig. X3 main effect, when it really shouldn't.

Is there anything that can be done to the data (i.e. transformations) that will reduce collinearity beyond centring? Failing that, is there an alternative to HMR that comes mind, that may be better suited?

Ross Anon

I am not sure what you are still looking for. Your VIF scores do not look that bad AND they do not influence the power for the interaction terms, but only for the zero order variables/predictors. So, if you are primarily interested in the interactions, this wasn't an issue.

If all other assumtions look fine, residuals (normality, homoscedasticity), no outliers or influential cases, and linearity, you should use your results. It is how it is. Your model shows that you do not have enough evidence (except X1xX4) to reject the H0 for your regression coefficients. You may not interpret it as evidence for the H0, i.e. that the H0 is true or you regression coefficents are zero. Did you do an a priori power analysis? What about the regression coefficients itself? Although not significant, they may help understand your results. In case of too low power, gather a new, larger, appropriate sample size.

Thanks Rainer Duesing . You've been incredibly helpful.

Unfortunately, I didn't conduct power analysis. I'm really new to all of this, so I didn't know it was expected. Instead, I used a similar number of participants as the original authors (my study is looking at someone elses, and seeing if the effect they observed was moderated using my IVs). I'll certainly do so in future though.

This may be a silly question then. If the results so far are correct, and I find that none of the Xs are significant moderators (except X1X4 in Model 2, which had a non Sig. F Change). Would I be required to explain the significance of X3 in Model 1? From what I understand, it's only significant because it's in the presence of X1 and the other zero order variables / predictors. But as I've suggested it may moderate X1s relationship with Y, should I explain it outside of its moderating effects as an additive? Would I conduct separate simple regressions on X1 and X3?

In my opinion, you do not have to explain something for which you didn't have a hypothesis. As far as I understood it, you only had hypotheses for X1 and the interactions, right? You have to include the zero order predictors to correctly specify the interactions, but you do not need hypotheses for them.

That's like music to my ears. Thank you so much Rainer Duesing .

You are welcome. Good luck!

You didn't indicate your sample size and that may well be critical because it affects statistical power. The number of cases needed goes up with the number of predictors and interactions constitute predictors just as main effects do. Green (1991) offers useful rules of thumb on sample size selection in multiple regression which gives separate rules of thumb for the multiple correlation and individual predictors. Also, a stats text by Hinkle, Wiersma and Jurs (2002) suggested that it's rarely useful to use more than seven predictors in a multiple regression.

An afterthought: Statistical power for the detection of interactions is weak relative to that for main effects. Brian O'Connor at UBC has developed SPSS and SAS programs which deal with that problem, partly by substituting bootstrapped significance values for those based on standard statistical tables. See https://people.ok.ubc.ca/brioconn/orm/orm.html