It's really difficult to answer your question without more details about your experimental design, your sample size, the exact nature of your variables and so on, because there are so many ways for such an analysis to be over-simplistic...

For instance, in your presentation of the experiment, it's not even clear why you nead a *repeated measure* ANOVA !

The mention of SD suggests the referee is concerned with the underlying assumption of variance homogeneity (that is, the predictor changes the expectation of the predicted variable, but nothing else, especially not its variance). In some situations, assuming variance does not change is simply irrealist, especially when there is a known relationship between the two (log-normal data, Poisson data, binomial data and so many other situations except Gaussian!)

Thank you Emmanuel Curis for your help! (Un grand merci!)

Below are details about our sample size, the nature of the variables and the experimental design as well as how we developed our findings section (sample). Hope this could help you better situate the issue the reviewer is pointing. (merci par avance!)

Two studies were conducted in this research work:

study 1: testing the effect of one condition (condition 1)

study 2: testing the effect of three conditions (same condition 1, condition 2 and condition 3)

Design of the experiment:

study 1: we measured the perceptions before and after the manipulation, therefore repeated measure anova was requested as we are interested in the perception modifications.

study 2: we measured the perceptions before and after the manipulation for three conditions, therefore a mixed between-within subjects ANOVA was conducted to assess the impact of three conditions (condition 1, condition 2, and condition 3) on participants’ scores for Variable 1 across two time periods (before and after).

All variables are continuous variables (except for the experiment condition: 3 conditions therefore nominal variable). Exploratory analyses were conducted to assess the reliability and validity of the variables. We checked the multicollinearity, linearity and normality assumptions for each variable (i.e. Skewnezz & Kurtosis z-values scored less than │3.29│ (Tabachnick & Fidell, 2007), Visual check of Histograms, Normal Q-Q plots and Box plots). We then performed exploratory factor analysis to evaluate all items and constructs used in this study. To aid in our interpretation of the measured components, we did an oblique rotation on all of the measured items, and checked the data for the cross-loading items and those with communalities of less than 0.5. All items loaded on their respectful dimensions.

Sample description

the data analysis is based on 115 completed responses for study 1 and 301 respondents for study 2. For the two studies, this is beyond the minimum number of participants required by an a priori power analysis (G*power: Faul and Erfelder 1992) based on previous literature (given α = .05; Power (1-β)=.8 and expected effect sizes estimated from previous literature), In order to check for non-response bias, we conducted a two wave analysis test on collected data as suggested by Armstrong and Overton (1977). In this regard, we compared the sample demographics , sample characteristics as well as major findings of early and late respondents (these last are expected to have a similar profile to non-respondents). No significant differences were reported between the two groups. We could conclude that non-response bias was not a major concern in this study (Johansson, Raddats and Witell, 2019).

In the findings section, here an example on how we reported the findings

Study 1: There was a significant decrease in Variable 1 from Time 1/before (M=5.689, SD=1.008) to Time 2/after (M=5.411, SD=1.244), t(114)=–3.629, p=.000 (two-tailed). The mean decrease in Variable 1 was –.278 [95% CI: –.430 to –.126]. The eta squared statistic (.103) indicated a large effect size (Cohen, 1988, p284).

Study 2: There was a significant interaction between the three conditions and time, Wilks Lambda=.901, F (2,298)=16.398, p=.000, η2=.099. There was a substantial main effect for time, Wilks Lambda=.939, F (1,298)=19.350, p=.000, η2=.061, showing, on the one hand, a reduction in Variable 1 for Condition 1 and Condition 3 and, on the other hand, an increase in Variable 1 for condition 2. A follow-up test with t-test was conducted There was a statistically significant decrease in Variable 1 for condition 1 and 3. The magnitude of the decrease had a large effect and was higher for contion 3 (η2=.259) – than for condition 1 (η2=.139). As for condition 2, a paired sample t-test confirmed that the increase is significant and had a moderate effect size (η2=.04).

Thanks for the details!

I'm not clear on one point: you speak both of a single, continuous explained variable, and of multiple variables that seem to be answers to a questionnaire with items, hence basically discrete if not even qualitative. And also, if you have a single variable, then Wilks test seems unusual to test for the interaction, whereas it makes sense if you test simultaneously several variables, but that's much more complicated...

I guess you made dimensionality reduction on all items and did the analysis for the main component, or something like that? If so, that opens a huge set of questions for the the comparison of the first component between and after. But no need to detail them if I misunderstood what you are doing... Note also that I am far from a specialist of analysis items, so my advices are just generic.

The second question is: why use a repeated measure ANOVA, and what exactly do you call a repeated measure ANOVA?

« Why », because with only two times, it is much simpler to compute the difference and study it directly (« paired T-test », for study one, does exactly this; and for study 2, that would lead to a simple 1-way ANOVA of these differences), and application conditions are also easier to check, since they apply on differences only. Note that mathematically, this is the same model than repeated ANOVA, so if you have concerns about doing differences (thinking, for instance, that ratios are better, or that initial value may interfer with the difference size or...), then you should have the same concerns with the repeated measure ANOVA analysis...

A common reason not to use differences is the problem of missing values, but if you made dimensionality reduction you probably already get rid of them, and see also point below.

« What », because repeated measure ANOVA makes me think about the sum-of-squares decomposition approach, with a lot of assumptions on error distributions (but that may be a personnal bias), more than the maximum-likelihood mixed-models approach (with quite different assumptions). But only the second one allows handling missing values correctly...