The trouble with the design is that you don't know how typical the raters are. Raters who are involved in studies are often better trained, better motivated and more painstaking than typical raters. They may also share a working environment, which can mean that they agree with each other but may not agree with other similar raters from somewhere else.

I analysed a study with six raters. I was amazed to find that there were three pairs of raters who agreed well with each other, but agreement between the pairs of raters was poor. I was told that the study had recruited three pathologists and their trainees! So the trainee agreed with their pathologist because that's what they had been trained to do.

So no simple answer.

Yes, if you calculate the interrater reliability between the two raters.

Hello Béatrice, I´m not sure I understand. I think (as in RCT) the sample ist too small to extrapolate it to the real world.

The trouble with the design is that you don't know how typical the raters are. Raters who are involved in studies are often better trained, better motivated and more painstaking than typical raters. They may also share a working environment, which can mean that they agree with each other but may not agree with other similar raters from somewhere else.

I analysed a study with six raters. I was amazed to find that there were three pairs of raters who agreed well with each other, but agreement between the pairs of raters was poor. I was told that the study had recruited three pathologists and their trainees! So the trainee agreed with their pathologist because that's what they had been trained to do.

So no simple answer.

Science is not showing the absolutely truth it only increases the likelihood to find the truth.

The last comment reminds me of a saying from the scientific philosophy by Karl Popper, exemplified by e.g. the following: You can not prove that all ravens are black. Even if you find a thousand black ravens you still haven’t proven that all ravens are black. But if you find one white raven you have proven that all ravens are not black.

hello Björn, Popper referred to swans, but that is beside the point. Béatrice is right of course, it is about probability, and if you have only have 2 swans to refer to, the likelihood that your extrapolations are true are rather small. So I´m looking for a statician who can answer this little quest: how many assessors do you need to make a likely statement of the intertester reliability of a test.

Hello Ronan, it is a difficult question, but I think it could be answered by staticians, a group too which I definitely not belong.But my common sense tells me, 2 are not enough. I also suspect that the assessors are also the developers of the test, so there might be bias. If you create a bigger group of testers it is more likely that you will even the problems you raised.

Rónan is a statician and he already gave a good answer!

Björn, what about they gray ravens then:)

I think the number of desired raters depends on the confidence level in your results you want to achieve and the margin of error you are willing to tolerate.

Aside from the statistical considerations mentioned by Kirk, the real questions are non-statistical, and no statistician can answer them (but any statistician should ask them!).

What hypothesis is being tested? Are two raters adequate to test the hypothesis? And if so, how should they be sampled?

Without knowing the precise question being asked, no statistician can give you a helpful answer.

Hehe... in my pedagogical theory and science philosophy classes back in 1994-1995 the main example mentioned connected with Karl Popper was in fact with a thousand black ravens and the possibility of finding one white raven... hehe... and not a bird blossoming into a white swan when growing up... which reminds me about the fairy-tale “The Ugly Duckling” by H. C. Andersen, where the duckling in fact turned out to be a beautiful white swan having being bullied for being different from the other ducklings when growing up among ducks without knowing he really was a swan (a story still actual today - unfortunately one may add). Nevertheless, using a raven sounds more logical to me as most ravens are black... hehe... and many with various shades of grey yes (there is no black or white just different shades of grey - from the movie Soul Man if I remember correctly)... e.g. also the crows... where it may be very difficult to actual find a white raven... whereas both white and black swans are common. Well, I haven’t checked the original sources by Karl Popper here, so I may have missed or forgotten something...(!) ;-)

About the uncertainty, I may mention one issue here even if I don’t know if it would be of relevance for this specific case. When measuring something we calculate the mean value, and the uncertainty as the standard deviation of the mean. Naturally, this uncertainty is decreasing with the number of measurements. Noteworthy, there is also an uncertainty in the uncertainty, which is approximated as the following for a normal distribution (n = number of measurements):

Uncertainty of uncertainty = 1 / squareroot(2(n-1))

(given between 0 and 1, multiply with 100 % for percentages)

From this expression it may be seen that the uncertainty of uncertainty is approximately:

n = 2: 70.7 %

n = 3: 50.0 %

n = 4: 40.8 %

n = 5: 35.4 %

n = 6: 31.6 %

n = 7: 28.9 %

n = 8: 26.7 %

n = 9: 25.0 %

n = 10: 23.6 %

n = 15: 18.9 %

n = 20: 16.2 %

n = 30: 13.1 %

n = 50: 10.1 %

n = 100: 7.1 %

n = 500: 3.2 %

n = 1000: 2.2 %

n = 10 000: 0.7 %

Hello Ronan, tested in this case was a multi-layered assessment, which is used to test the level of conscious state in coma patients.

Hello Björn, is there any way you could produce such a nice little array of numbers concerning the number of Raters needed to ascertain a certain amount of confidence level Kirk mentioned afore. Seeing the equation I´m rather sorry I did drop out of my mathemathicscourse ages ago. As far as Poppers swans are concerned https://www.absolute-strategy.com/x/black-swan.html . I do have a couple of books by his hand (mostly the popular he wrote, only one of the really deep going philosophical works), but the search function isn´t as handy as google. Happy new year to all of you!