Hello and thank you in advance for your help.
Is there any rule of thumb for the number of trials you need in an experiment?
I have a task I am piloting. It has 2 conditions and 10 trials for each condition. Thus far, I have 38 participants and I am already seeing a pattern that points to an overly difficult item. With the data I have thus far, analysis shows that item is significantly harder than the easiest item. I'm wondering if it is better to replace that item or remove that item. I ask because the task does seem to be causing some stress in participants as they are competing against themselves.
Hello Kym,
Part of the answer may hinge upon whether your study is concerned with respondents' score trajectory over trials.
I'm assuming that item = trial for this discussion. The answer depends on how you had intended to treat individual item/trial outcomes. If treated as a single score (summed value across all items or trials), then a change from 10 to 9 elements in the sum is unlikely to result in a substantive change. Note that, for summed scores, the principal impact of the number of elements comprising the sum is that of the reliability of the total (or averaged) score across items/trials.
If, on the other hand, you are using items/trials as a repeated measures dimension, then: (a) the number of trials and (b) the correlation among trials will have an impact on the statistical power of the test of trials (as a main effect) and of treatment by trials interaction (which would be the test to reveal whether groups had different profiles across items/trials). Again, however, changing from 10 to 9 is unlikely to make much of a difference. You can run "what if" scenarios in the software, G*Power (available free at: http://www.gpower.hhu.de/) to get a more precise answer to the question of how much of a difference.
Good luck with your work!
It is not necessarily how many trials are to be conducted, but rather the number of individuals in a trial. One trial can be conducted in the world's population as study volunteers and the world's population is large enough (in most cases) to get statistically significant results upon analysis of the date from this population.
I find this useful:
Gerald van Belle: STRUTS: Statistical Rules of Thumb
www.nrcse.washington.edu/research/struts/chapter2.pdf
Thank you Dennis Mazur and Jochen Wilhelm. I greatly appreciate the help. Thank you David Morse that really gets at what I am doing. Part of my issue is that each trial delivers an accuracy score, allocated time, and actual time. So the accuracy score is summed but the others are used in ratios or averaged. I was actually deleting trials and rerunning the statistics to see if it made a difference, but then realized I have another problem - I have ceiling effects in accuracy on several trials. So, apparently I won't be replacing the difficult item, but will be replacing the easier items. I will run scenarios in G power to see how much I can reduce trials since the task will now be more difficult. Thanks very much for that suggestion.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression