One primary factor in tests of statistical significance is sample size. With a large enough sample, even small effects can yield small p values leading to "significant" results. One way to convince the reviewer may be to conduct a power analysis (e.g., in G*Power) to show that with the sample size in your study and the given effect sizes, your power was high (so that all effects were found to be statistically significant).

Christian Geiser Please give a more detailed explanation, Thank you.

When you google "post hoc power analysis," you should find plenty of tutorials/examples. I generally do not recommend post hoc power analyses (the primary purpose of power analyses is sample size planning--by definition an a priori procedure). However, in your case, a post hoc power analysis may be useful to show that your study was sufficiently highly powered. That way, you can demonstrate that in was "adequate" or "plausible" that the effects found in your study were "statistically significant."

Christian Geiser I did it from the Design Expert software, which does not have the ability to do this method.

It is trivially not true that all effects being p < .05 is (in general) evidence of data fabrication. The true effects could be large or the sample size provide adequate power to detect smaller effects (or both).

For example if the statistical power to detect a true effect is .95 then the probability of getting p < .05 for three effects in a 2x2 ANOVA is .95^3 = .857. Of course we don't know the true effect she, but this illustrates that is could be quite common to see all effect significant in a well powered study.

What about a more common scenario with .80 power? That's still a 50:50 occurance? However if it's for a more complex design that's less plausible - for instance for a 2x2x2 design it would be about a .21 probability .(because there are 7 effects). So it's less plausible that a study planned for .80 power would find all significant effects as the number of effects rises.

However, this assumes that the reviewer is concerned simply with p < .05. A more legitimate concern is p values that are just significant. i.e., p = .01 to .05, this can be a sign of issues with data or analysis because it really isn't that likely at all. There are published papers on this (e.g., see this blog https://www.the100.ci/2018/02/15/the-uncanny-mountain-p-values-between-01-and-10-are-still-a-problem/ ).

e.g., if you report a 2x2 ANOVA ad the three effects are p = .041, .032 and .023 that would be an unusual patterns. p values aren't that stable.

Nevertheless even if the pattern is improbable or suspicious that doesn't mean the data are fabricated. The data could have unusual distributions (e.g., discrete values with restricted range) which means that only a small number of p values are possible. This might indicate a problem with the statistical model - not the data. There could be errors in the analysis (e.g., data that are uncorrected coded or labeled) - though that's a bit more unlikely. You could also just be lucky/unlucky (depending on how implausible the pattern is).

It is not uncommon to have all significant p-values in an ANOVA if the study is well-designed, has high statistical power, and uses relevant variables. Significant results can naturally occur in robust experiments with a strong effect. However, it's important to ensure proper data collection, sampling, and assumption checks. Significant p-values alone do not indicate data fabrication, especially if they align with the study's context.