Hello Rogier,

For me, the answer depends on what your specific research aim might be.

1. If you're interested in estimation of population parameters (e.g., the levels of the 10 variables that your raters will judge), then this is a sampling precision question, the the usual advice would be to start with the ordinary simple random sample size type formulae, which depend on: (a) risk of being wrong in your estimation, and the associated degree of confidence for any interval estimate; (b) the target degree of precision for the parameter estimate; (c) the population size (I presume this would be essentially infinite) relative to the sample size; and (d) the amount of variance associated with a given variable.

Higher variance = bigger sample; Lower risk = bigger sample; infinite population size = Somewhat bigger sample; Higher precision = bigger sample. The most variable of the 10 lesion attributes should be the one to guide your sample size determination. Here's a sample on-line calculator for this sort of thing: https://stattrek.com/survey-sampling/sample-size-calculator.aspx Note that number of lesions to be rated is the "sample" here.

2. If you're interested in how much or how little consistency there is among judges, then I suspect 12 should be satisfactory. If you're treating judges as a random effects variable (e.g., randomly selected from pool of available judges), then usually more levels is preferable to fewer levels. There are a number of indices that quantify the degree of rater/judge consistency across a set of samples. These include: (a) extensions of Cohen's (Fleiss's) kappa; (b) the Kendall coefficient of concordance; (c) Generalizability theory (which estimates relative variance in scores attributable to specific sources, such as raters: see this link for an introduction: https://books.google.com/books/about/Generalizability_Theory.html?id=gzyKRZWrm9EC

3. If you're interested in estimating the correlation/consistency among judges' ratings within a specific amount of accuracy, then this essentially becomes a sample size needed to estimate a correlation within a specific level of precision. So, it's like #1, except that the standard error is different (see this link, which treats precision as the difference between a null and alternative hypothesis: https://core.ac.uk/download/pdf/268085221.pdf).

4. Finally, if you decide that the number of lesion samples is too high for any one judge to rate (on 10 attributes), you can randomly assign samples to judges. If you like, you can explicitly assess judge "effect" in such matrix sampling designs by anova/generalizability theory/regression models.

Good luck with your work.

Krippendorf's alpha is good coefficient when you have multiple raters on multiple variables.

I think it is calculated by dividing the smaller total count observed (from one observer, relative to the other) by the larger total count (from the other observer).