I really don't follow your question but if interaction is what you want to study you need some type of factorial Experimental design. On second thought since you have data already I would put it aside and get a copy of Montgomery's Design and Analysis of Experiments and design an appropriate factorial Experiment. Then collect data and do the analysis as the book suggests. I see no way for you to get anything useful from your current approach. Try the z-library for some useful references. Best wishes, David Booth

Fabian, it's a little difficult to give an answer without knowing the aim of the experiment i.e. the hypothesis, but what I understand from your question is that you have structured data in the form of 5 task rounds x 3 subject groups x 2 genders and you are analysing task performance. I don't know what distinguishes your subject groups, so I don't know what sort of comparisons or conclusions you can make there. You seem to have made the right decision about using non-parametric ANOVA analysis. Your question seems mainly to be about cherry-picking task rounds for group comparison. The answer to that is that there are two effects usually at play in repeated task performance. One is learning or skill-building, which normally results in improvement in task performance. The other effect is fatigue, which normally results in deteriorating task performance. You haven't mentioned whether the 5 rounds of task performance are contiguous or separated by hours, days, weeks etc, so it is hard to know what role fatigue may play. If the task repeats are separated by long intervals, forgetting may also become a factor. For all three effects however, you need to consider the progression of task performance throughout the five rounds and try to identify trends in that progression. The best way to do this is to plot means and confidence intervals on a timeline. This shows exactly the same information as ANOVA, but it is in a form that is much easier to understand, particularly for non-specialist readers. You can then decide whether you can model task performance changes by linear methods, by a nonlinear fit or whether the progression is random. Just picking trial pairs for comparison is not very meaningful.

Hello Ian Dash

Thanks a lot for your reply!

You are correct. Let me add a bit: I have three groups: a control group, a monetary treatment, and a non-monetary treatment. I am comparing the performance across the groups, but naturally, e.g. gender and others are interesting aspects to investigate. The groups are, more or less, directly after one another, so there is both learning and fatigue. However, either there is nothing of the two or they balance each other out since there is no significant change over time.

I do not want to do cherry-picking, I just heard a post-hoc helps to identify the real changes. I do have a graph comparing the progression between genders and I want to use it (see attached file). Could you confirm my understanding of it?

[Please note that the legend is wrong, blue should be male and red female]

As far as I see it there is no significant change for females, while there is a change for males (also viewable in the tables). Although it is not huge, there seems to be a learning effect that outbalances any fatigue effect for males. Would you confirm that or would you say that the effect is too small to be interpretable and is more likely a random occurrence?

Thanks a lot, I really appreciate your help!

Fabian, yes, your plot looks like what I was suggesting except for formatting. Error bar plots are usually formatted like these https://scc.ms.unimelb.edu.au/resources-list/graphs-for-statistical-analysis/graphs. The last one on the page is probably the best version for your purposes.

There is often some confusion about how to interpret error bars. A common mistake is to assume that there must be no overlap between the bars to have a statistically significant difference. This is incorrect. Statistically significant difference means that the mean of one distribution is outside the confidence interval of another distribution. You have a statistically significant difference between your two gender groups at round 5 (but only just). You also have a statistically significant difference for the males between round 1 and round 5. There are other conclusions you can draw about differences, but I'll leave those to you.

Since I don't know the difficulty or the duration of the task, I can't comment on fatigue effects except to say that learning effects appear to outweigh any fatigue effect for the males, while for the females the two effects appear to balance out or else neither learning nor fatigue are occurring, since their performance is fairly constant.

A simple way to express the performance trend would be a linear regression of the means (or medians if you are using nonparametric analysis). Your results don't look like they need a nonlinear regression. The regression line for the males will have a definite slope, while for the females it will be pretty close to horizontal.

You may also find it useful to plot individual performances of subjects over the five rounds (take care to preserve anonymity of course). That can show whether in the female group there are some learners and some subjects affected by fatigue, or whether everyone was consistent all the way through. It could also show in the males whether you have some strong learners and some non-learners or whether everyone is learning at about the same rate.

PS if a reviewer challenges you on using an error bar plot (it has happened to me) you can cite this reference to justify the technique: Article The Relation between Confidence Intervals and Tests of Signi...

PPS you may also find it useful to plot frequency histograms for each subgroup and each round and overlay them when making comparisons. Sometimes patterns can be seen this way that are not visible from numerical tests. This is usually too much detail to put into a paper, but it can be useful for understanding the fine detail of your data.

On further reading, it seems there is a group of statisticians / researchers who prefer confidence interval analysis to Null Hypothesis Statistical Testing - see The earth is round (p < .05), Jacob Cohen, American Psychologist Dec 1994, pp 997 - 1003. My own view is that confidence intervals are just another way of presenting the same information - I don't believe the two are mutually exclusive. Cohen makes interesting and amusing reading though - he even introduces a new acronym: the full name is Statistical Hypothesis Inference Testing. 🙂

Thanks a lot Ian Dash ! You helped me a lot, I will dig a bit deeper into it! :)

Here's one more reference on confidence intervals and p-value testing. This one is a bit more balanced than Cohen. Article Confidence Interval or P-Value? Part 4 of a Series on Evalua...

I'll give this a rest now unless you have more questions, but will leave you with a final piece of advice: if you want to keep reviewers happy, include numerical ANOVA results as well as any confidence interval plots. If I were reviewing the work I'd be happy with just the confidence intervals, but I know there are reviewers out there who would not be. Good luck with your work.

Thanks Ian!!

Attached you will see the plot I made (marginsplot). I would be very happy for your feedback on it. In any case, thanks for the great help and for the resources! Is very nice of you. I also looked into the recommendations you gave me and I have to figure out a way to plot them, but looks promising :)

Fabian, you've done a nice job with the confidence interval plot. If you can find a way to separate the bars sideways just a little it will be even easier to understand. Then you need to overlay the regression lines and the job is done (apart from writing the text to explain it all 🙂).