Dear Moi Iannello

I am not very familiar with CRSE, but what I found is that is not very recommended, since it underestimates the standard errors.

Nevertheless, I would not agree with your approach and idea that something is "robust" or "not robust" if the analyses agree or not agree, respectively. All three approaches have different assumptions, test different statistical hypotheses and are either omnibus tests or not. Therefore, I find this approach odd in the first place. I think you cannot "verify" one result by conducting a completely different analysis, albeit on the same data.

You also did not tell, how you accounted for the clustered data in the ordinal regression approach and what kind of ordinal regression you used.

Thank you for your reply.

To address the limitations of IBM SPSS, which has a restricted range of robust methods (Field, 2018), Huang's (2020) SPSS macro was utilized. This macro was specifically designed to provide cluster-robust standard error (CRSE) estimates, which is crucial for analyses in which observations are not independent of each other. Using this macro, the aim was to adjust the standard errors to account for the clustering structure of the data at the individual level, ensuring more reliable results in the analysis.

I also conducted ordinal regressions, using GENLIN procedure on SPSS, to account for the dependent variables' ordinality. This type of regression seems well suited for my study, as dependent variables are also ordinal. The participants' responses in this study were based on a 7-point Likert scale, ranging from "strongly disagree" to "strongly agree".

My goal was to : compare the results obtained from the Wilcoxon signed-rank tests with the regressions with clustered standard errors at the individual level to the results and as a confirmatory analysis use the the ordinal regressions to ensure that the conclusions about the impact of different scenarios on the dependent variables are consistent. The goal was to check for the robustness of the Wilcoxon test results. If the conclusions from the different tests diverge, it suggests that the observed effect is less reliable. Conversely, if the effect is significant in all three tests, it is robust.

This approach was recommended to me by a teacher, but the problem was that I didn't quite understand how to analyze the fact that some results were significant in wilcoxon tests and CRSE regression but not ordinal regression. Even if this approach seems strange, do you know how I could interpret this? Is it because ordinal regression didn't take grouped variables into account?

I do know how to do an ordinal regression with clustered data on spss...

Dear Moi Iannello

really, my personal opininion is that the whole approach does not make any sense at all.

1) The different models test different statistical hypothesis, therefore, even IF all three approaches would show a significant result, this does not mean that the results converge, nor that the general result is robust. You cannot conclude that, its like comparing apples with pears.

2) You state yourself that the DV is ordinal, in that case using an approach for metric variables, even if you account for clustered data, is not optimal and again, tests something different than the Wilcoxon test.

3) Which ordinal approach did you use, logit or probit regression? I never used it with SPSS, therefore, I do not know the options there.

4) Again, you stated yourself, that you did not account for the clustered data with your ordinal approach. How do you think is the result comparable to the other two?

5) You say: "I do know how to do an ordinal regression with clustered data on spss" Did you mean that you dont know how to do it? If yes, why did you do it in the first place, if you already knew that it is the wrong approach? I am really puzzled..... basically, you knew from the beginning that 2 out of 3 analyses are not suitable for your data (one is not for ordinal data and the other does account for clustered data), what do you expect will the results tell you? Garbage in, garbage out.

Without knowing more, I would use a probit regression, to account for the ordinal data structure. The model assumes a normally distributed latent variable (it uses the normal CDF instead of the logistic CDF, but makes the interpretation easier), which seems reasonable, if you already used a OLS regression and thougt it will work. To account for the clustered data (repeated measures), I would use a linear mixed model (aka. multilevel model). I dont know if this is possible in SPSS, but surely in R with the ordinal package (I havent used it so far, but is capable of cumulative linear mixed models and uses the lme4 syntax) or go Bayesian (I would recommend) and use the brms package, which also uses the lme4 syntax.

I assume Moi Iannello is talking about the macro described here:

Article Using cluster-robust standard errors when analyzing group-ra...

https://www.youtube.com/watch?v=m_GHzXUaI_Y