I am running a study involving N=100 items that subjects will compare, side-by-side, and declare the preferred one.

This process is known as Paired (or Pairwise) Comparison: https://en.wikipedia.org/wiki/Pairwise_comparison

Let N=100 be the number of items to vote. It leads to P=100*(100-1)/2=4950 possible pairs to vote.

Let's call S the number of subjects that will take part in the experiment. I can imagine that reliable results can be claimed with S>=30, with every subject voting on all the 4950 pairs.

However, I can not pretend to ask 4950 votes to each subject!

Differently, I want to fix to V=10 the number of pairs that each subject vote. If I aim at retrieving 30 votes per pair, the number of subjects needed is S=30*4950/10=14850 = TOOMUCH!!!

How many votes are needed to claim believable results? Is there any way to reduce this number?

I am using the prefmod R module for my experiments.

https://www.jstatsoft.org/article/view/v048i10

But I don't know how to check for an indicator of good "statistical significance" for the results collected so far.

Similar questions and discussions