I am running a study involving N=100 items that subjects will compare, side-by-side, and declare the preferred one.
This process is known as Paired (or Pairwise) Comparison: https://en.wikipedia.org/wiki/Pairwise_comparison
Let N=100 be the number of items to vote. It leads to P=100*(100-1)/2=4950 possible pairs to vote.
Let's call S the number of subjects that will take part in the experiment. I can imagine that reliable results can be claimed with S>=30, with every subject voting on all the 4950 pairs.
However, I can not pretend to ask 4950 votes to each subject!
Differently, I want to fix to V=10 the number of pairs that each subject vote. If I aim at retrieving 30 votes per pair, the number of subjects needed is S=30*4950/10=14850 = TOOMUCH!!!
How many votes are needed to claim believable results? Is there any way to reduce this number?
I am using the prefmod R module for my experiments.
https://www.jstatsoft.org/article/view/v048i10
But I don't know how to check for an indicator of good "statistical significance" for the results collected so far.
The first part of your question, about the sheer number of comparisons, might be answerable with the 1000 Minds approach co-developed by (a colleague) Paul Hansen (see http://www.otago.ac.nz/economics/staff/otago077900.html or https://www.1000minds.com/). This would greatly reduce the number of pairs that need to be evaluated by the participants in order to determine all (transitive) preferences. While I haven't used the approach myself, it has been successfully used by many researchers and the approach seems sound.
The second part, about sample size, could possibly be simplified to how precisely you want to estimate the preference proportions. For example, if you wanted to know what percentage of participants preferred A to B, you could estimate this with 95% confidence interval half-widths (i.e., plus/minus) 10% with n=104 (exact binomial confidence interval under the worst case of 50%). If you could accept less precision (say, plus/minus 15%), then n=48 would suffice. You could get these preferences through asking n=104 (for plus/minus 10%) participants about each comparison of interest, or you could rely on transitivity (as with 1000 Minds) and ask them fewer questions such that, by assuming transitivity, you now have all 4950 preference combinations on each participant.
Alternatively, you could power on detecting differences in preference for a combination of A and B. For example, to have 80% power to detect a preference of 65% or higher against the null of no preference, so 50%, using a two-sided binomial test at the 0.05 level would require n=90, assuming no ties.
You could possibly do something around the precision of rank orders and use simulations to determine how many participants/responses you'd need to be able to estimate rankings in this way, but the above suggestions would be much simpler (both to do and to explain to others).
Hope that this is of some use in your research.
The first part of your question, about the sheer number of comparisons, might be answerable with the 1000 Minds approach co-developed by (a colleague) Paul Hansen (see http://www.otago.ac.nz/economics/staff/otago077900.html or https://www.1000minds.com/). This would greatly reduce the number of pairs that need to be evaluated by the participants in order to determine all (transitive) preferences. While I haven't used the approach myself, it has been successfully used by many researchers and the approach seems sound.
The second part, about sample size, could possibly be simplified to how precisely you want to estimate the preference proportions. For example, if you wanted to know what percentage of participants preferred A to B, you could estimate this with 95% confidence interval half-widths (i.e., plus/minus) 10% with n=104 (exact binomial confidence interval under the worst case of 50%). If you could accept less precision (say, plus/minus 15%), then n=48 would suffice. You could get these preferences through asking n=104 (for plus/minus 10%) participants about each comparison of interest, or you could rely on transitivity (as with 1000 Minds) and ask them fewer questions such that, by assuming transitivity, you now have all 4950 preference combinations on each participant.
Alternatively, you could power on detecting differences in preference for a combination of A and B. For example, to have 80% power to detect a preference of 65% or higher against the null of no preference, so 50%, using a two-sided binomial test at the 0.05 level would require n=90, assuming no ties.
You could possibly do something around the precision of rank orders and use simulations to determine how many participants/responses you'd need to be able to estimate rankings in this way, but the above suggestions would be much simpler (both to do and to explain to others).
Hope that this is of some use in your research.
Dear Fabrizio
I agree with Andrew for the first part , and for the estimation of sample size for each combination, we know that sample size depends on standard deviation ( from previous studies ) and the margin of error you decided .
G*Power software is effective tool to calculate sample size for many ranges of experiments. Also, you can determine effect size and power of the test, G*Power is free to download and easy to use after reading the manual, the download link:
http://download.cnet.com/G-Power/3000-2054_4-10647044.html
The link to download G*Power manual:
http://www.gpower.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Psychologie/AAP/gpower/GPowerManual.pdf
Good Luck
- Badges
- Science topic