Marco Marini While I certainly can't know the reviewer's intentions, I can venture a guess as to what they're getting at. Think of the question rephrased as: How did you determine the number of trials to use?

A good researcher should have an answer to this question - backed up either with a priori data, or at the very least, cite previous evidence that justifies the included number of trials. Number of trials should never be arbitrary, and for all the reviewer knows the number of trials was arbitrary. The argument 'We found a strong effect size and small p-value with our number of trials, thus our number of trials is fine' is a poor argument for two reasons - one theoretical and one practical. The first is that, while these parameters worked for this study, it doesn't really teach us anything about future research on this topic (i.e. replication). The second reason is that effect sizes and p-values can fluctuate pretty dramatically when you manipulate the number of trials (especially when there are as few as were used in this study). As David Eugene Booth suggested, it's possible you got lucky. Maybe increasing the number of trials to 40-50 per condition would have greatly reduced the effect size and increased p-values due to regression to the mean (or something like this?)

To reiterate, a statistically-inclined reviewer is not going to be persuaded by observed power. Statistically, I would emphasize the effect sizes rather than observed power or p-values, try to cite research on this topic that can help justify the study parameters, and definitely include this as a limitation in your paper's Discussion.

Best,

Blaine

Alessandro Ansani I'd forget about observed power. This statistic is rather misleading and won't convince most reviewers. Instead, I'd focus more on the observed effect sizes - which provide a better basis for defending the findings.

Since any power analysis at this point would be conducted post-hoc, the best advice I can give is conducting a power analysis using Monte Carlo bootstrapping simulation (or something similar) using the effect size you found and varying the number of trials in the simulation. The output will be a power curve showing power as a function of number of trials. Unfortunately, I've never done such an analysis in SPSS, but I've attached a tutorial on how to conduct such an analysis in R using the 'simr' package.

*Note: since the data are non-normal, perhaps ANOVA isn't the best option? It might be worthwhile to analyze the data using a Generalized Linear Model instead? This way you can avoid the hassle of all the additional tests (Kruskal-Wallis, etc,)

Best of luck!

Blaine Tomkins raises a good point but if the editor agrees with reviewer which probably will be the case , you will need to repeat the expt. so look at both the effect size and power for the next time around. Always do this stuff as part of your experimental design process. Best, David Booth

A nasty question (that shouldhelpto resolve the issue after some thorough thinking):

Why, essentially, do you (or do we, in general) test statistical hypotheses?

David Eugene Booth What do you think about conducting a post-hoc sensitivity analysis? I've heard this from reviewers in the past.

Blaine Tomkins It can't hurt but that isn't the point the reviewer and Jochen Wilhelm are making. We do the test in order to control the probabilities of making a type I or II error. If we mess that up depending on our field someone may die or fail to be as well as possible because we screwed up the experiment when if we got it right right we help somebody. I wouldn't want a paper like that in my journal if I could help it. IMO get the errors controlled so nobody is sick needlessly. Sensitivity analysis doesn't answer the same question. Do the stats right is my opinion and try to make life a little better Calculate an appropriate sample size and repeat the experiment. Get it as right as possible. That's the purpose not to add a pub to the CV. Best to all, David Booth

Hello Alessandro,

So, even with a conservative alpha level, you found significant results?

If you got statistically significant results, you obviously had sufficient statistical power to detect effects of the magnitude observed in your data set! That, by itself, should address what sounds to me like a misguided concern on the part of the reviewer. C'est la vie.

Power would only be a concern for instances in which you failed to obtain statistically significant differences. That is, someone might claim that the result was merely a type II error rather than an absence of effect/difference. Were this the case, you could do a simple, "What-if" analysis to determine how many more cases would have yielded a significant result, had the same degree of effect/difference been observed. While that doesn't guarantee anything for future studies, at least it might give researchers a clue as to what sorts of sample size might be plausible for studies of a similar nature.

As a general observation, the power-efficiency of the Kruskal-Wallis test is approximately 95%+ that of the one-way ANOVA when all of the parametric assumptions are satisfied. If distributional assumptions are not satisfied, then the K-W can be more powerful (and less likely to be biased); both rely on homogeneity conditions, however.

Good luck with your work.

I am going to at some risk disagree with my good friend Prof. David Morse . I agree with Prof. Morse' analysis of the result that the questioner in the end perhaps had the good fortune to have the requisite power to justify the conclusion reached. However good experimental design is not one of lucking out to have a the power required in the end but rather planning the experiment given that we do not know the results ahead of time. Since the result could have been different as we knew in the planning and design stage since if otherwise there would be no need to experiment.Thus, We should have planned to do an experiment that did not require a particular conclusion to be valid only when we write the paper. Further IMO there are a few other considerations. I will assume that the experimenters expected to do the analysis by means of an ANOVA or it's equivalent. It seems to me then the following conclusions hold.

1. We do not have a research question so I am assuming as I did above that a two sided test would be employed.If there was as a previous result that lead to a different hypothesis to be tested it should have been presented in the exposition given to the referee.

2. If I am correct in 1 then it would have been easy to use G*power or it's equivalent to compute an approximate sample size to generate that value of sample size that would suffice whatever conclusion obtained.

3. Given that non parametric tests were used that are a bit less powerful than the planned ANOVA it would behoove the experimenter to increase the say G*power n a bit to take into account this possibility.

4. Given that the result was as described which would have made power irrelevant I would have pointed that out to the reviewer.

Now everyone knows hindsight is always 20:20. But I think a better design would have taken more possibilities into account and would have made the reviewer's concern less likely. My old experimental designs professor used to say if it doesn't cost a lot take some extra data. If that would have been done here then it is unlikely this situation would arise. As my professor used to say: Proper Prior Planning Prevents Poor Performance. and sometimes extra work Anticipation and some luck are the experimenters friend. Best to all, David Booth

Hi guys,

I'm a co-author of Alessandro. We really appreciate your suggestions. However, we'd like to better understand this issue...

Your kind replies have focused on the observed power, whilst our problem is to justify the number of trials. We do not have to necessarily bring observed power into this (it was just our idea).

It still unclear on which basis the reviewer thinks that our trials are not enough (taking into account that our sample size is definitely appropriate and our p values point in this direction too).

To put it in a nutshell:

1) What prompts the reviewer to raise a question about the number of trials (even considering our significance levels)? Is it totally arbitrary (i.e., s/he would have liked to see more trials) or rather s/he has some valid reasons for claiming we did not have enough?

2) Can the power analysis (observed) justify the number of trials? And what about the effect size? If the number of trials is arbitrary, and our results are strongly significant, why should we increase trials? (especially with a good sample size)

thank you guys in advance.

Marco Marini While I certainly can't know the reviewer's intentions, I can venture a guess as to what they're getting at. Think of the question rephrased as: How did you determine the number of trials to use?

A good researcher should have an answer to this question - backed up either with a priori data, or at the very least, cite previous evidence that justifies the included number of trials. Number of trials should never be arbitrary, and for all the reviewer knows the number of trials was arbitrary. The argument 'We found a strong effect size and small p-value with our number of trials, thus our number of trials is fine' is a poor argument for two reasons - one theoretical and one practical. The first is that, while these parameters worked for this study, it doesn't really teach us anything about future research on this topic (i.e. replication). The second reason is that effect sizes and p-values can fluctuate pretty dramatically when you manipulate the number of trials (especially when there are as few as were used in this study). As David Eugene Booth suggested, it's possible you got lucky. Maybe increasing the number of trials to 40-50 per condition would have greatly reduced the effect size and increased p-values due to regression to the mean (or something like this?)

To reiterate, a statistically-inclined reviewer is not going to be persuaded by observed power. Statistically, I would emphasize the effect sizes rather than observed power or p-values, try to cite research on this topic that can help justify the study parameters, and definitely include this as a limitation in your paper's Discussion.

Best,

Blaine

I agree with Blaine Tomkins . I will also add that if you control both type I and II errors with an increased sample size you are less likely IMO to come upon a small p-value by chance because power=1-P(type II error) and you have controlled for that as well the Type I by the usual sample size calculations. Best, D. Booth

Hello all,

If the analysis was in any way a repeated measures design and if the individual trials were recorded as data points, then the number of trials would play a role in the df for various effects, most notably any interactions involving trials. So, in this way, they could have an effect on the statistical power (sampling distributions of F involving larger df have lower critical values).

However, if the authors simply summed or averaged the outcome over the set of trials and recorded that as the DV, then trials still has potential import for several reasons:

1. Too few trials would not yield as reliable an indication of participant performance as would a higher number.

2. Too many trials might result in participant fatigue or inattentiveness, either of which might bias the estimated performance downwards.

So, what's the "right" number of trials? In my opinion, unless one were willing to evaluate this specific task empirically using samples from the target population of interest, I don't think anyone would really know.

Best wishes to all.

Hello guys,

Thank you for all the replies. We totally see your point but maybe we have a new idea.

What do you think about a generalized linear mixed model including both random and fixed effects? In this way, we should be able to control for the excess variability in the dependent variable.

Since we had a mixed paradigm (with faces crossed with subjects) we could run the model with random effects for 1) subjects 2) Stimuli and 3) an additional interaction subjects*faces. If we find a significant association between condition and accuracy (in emotion recognition) net of random effects, can we solve the concern about the numbers of trials?

Or is it better to re-run the analysis with a monte carlo bootsrapping to simulate a larger sample?

Thanks in advance guys!

Alessandro Ansani and Marco Marini, you have an interesting experiment, the outcome of which can be understood in a relatively straightforward and intuitive way. Unfortunately, and somewhat surprisingly, making a convincing case for your hypothesis, even using the simplest formulation of the research question, is difficult. Moreover, undertaking a meaningful, interpretible, statistical approach is not trivial.

Imagine this simplest formulation. Subjects are shown images of people in face-masks and asked if they are showing emotion - yes/no. That would be the experiment. You would likely use the exact test of goodness of fit for a binomial variable (the "sign test"). The probabilities here are as simple as possible - the familiar coin toss setup. How many times would you flip a coin to decide it's fair?

You've designed an experiment with potentially multiple coins being tested, as well as predictions for how the ridges and and bumps on those coins will affect your result. And finally, instead of dealing with exact likelihoods of coin flips, your measurements are probably ordinal/nominal survey data that arguably cannot be parameterized.

Put this way, how large of a sample would one need to statistically argue that these coins are legitimately skewed (and their interaction attributes contribute to this somehow)? You would need a lot of coin flips! This is the crux of the so-called replication crisis brought on by reliance on statistical checklists and cookbooks instead of thoughtful design.

This is why I bring up Ockham's razor, though my use of it here is not one usually heard. Testing for multiple interactions and setting up needless blocking of presumably nuisance variables (beware 'table 2 fallacy') does not make the experiment more robust, but to the contrary, dilutes the information value of your sample, which is a resource not to be taken for granted.

If you are not confident enough in your results to argue convincingly for their validity to the reviewers and cannot do so on the merits of the experimental design, then I'm not sure what can be done. Post-hoc power is in my own and many others' opinion a meaningless procedure and often misleading. Even Gpower and the like turn empirical a-priori effect considerations into a one-size fits all matter of coin flips and marbles.

My advice would be to assess whether you can demonstrate your main effect. I don't have your raw data so I can't offer much more insight (I'm not a social scientist either). I agree with Jochen Wilhelm and David Eugene Booth that you may not have a sufficient n to test all the inferences in your analysis. Moreover, the trouble with a low n is that it may not be informative in establishing a null outcome either, were that route taken.

Randomization methods (monte carlo, bootstrapping) cannot "simulate larger samples" unfortunately, but a permutation test would be an excellent way of demonstrating that your results are likely not due to chance.

Good luck.

Just my two cents to this topic:

1) From a practical point of view, what would satisfy the reviewer or what is his concern? As far as I understood it, it is not about the sample size (you said it was sufficiently large), but the amount of trials you used to assess your dependent variable within each participant, right? If so, maybe the concern is about the reliability of the measure and the amount of your trials for it, like the amount of items in a questionnaire. This concern is valid and should have been addressed a priori (like Blaine Tomkins mentioned) to have a good argument for 10 trials. Maybe one solution or at least one argument is to show the achieved reliability for your data and the different conditions. If reliability is large, this would support partially the use of 10 trials.

2) If you would like to employ multilevel models, you have to be careful how to set up the levels and the nesting within each other. Is it a nested or a crossed design? As far as I understood it, I would guess it is crossed (subjects and items/trials are crossed), but please see the following discussion, this might help.

https://stats.stackexchange.com/questions/228800/crossed-vs-nested-random-effects-how-do-they-differ-and-how-are-they-specified

The design is a cross-classified one with photographs/faces and participants. Thus, you have two sample sizes. It seems the issue is with the number of photographs. As Rainer Duesing says, a cross-classified model seems appropriate and treat photographs as random effect, though this begs the questions of what these faces are a random sample of (but the same question exists for participants). The power relates both the number of participants and photos, so your power calculations should use appropriate statistical models for that.

Rainer Duesing brings up a good point. This might be considered a stratified sample with Strata the trials an d n your final sample size. There are programs to programs to deal with sample size issues for such samples. Should you wish to consider such an approach you might look at the stratified chapter in this:

https://b-ok.cc/book/2324520/acf6e2

Best, David Booth

Hi all,

Interesting problem here. In answer to the power question, I'd refer you to Marasciulo and McSweeny's text, Nonparametric and distribution-free methods for the social sciences (ISBN-13 : 978-0818502026). Inside you will find a chart on asymptotic relative efficiency of nonparametric tests when compared to parametric tests, and those data would enable you to calculate your power for the reviewer (have to do it by hand though). As I recall (and please don't rely on my memory) the greatest difference in the chart was 1.5, meaning you'd need 1.5 times the number of subjects for equal power when compared to the corresponding parametric procedure. The text is pretty old (1977) and a bit hard to find, your librarian could probably help you find it.

Best wishes on your study!

Rich